[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:07:16 dist-test-slave-hhpq krb5kdc[7246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:07:16 dist-test-slave-hhpq krb5kdc[7246](info): set up 2 sockets
Sep 27 15:07:16 dist-test-slave-hhpq krb5kdc[7246](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 15:07:16.996958 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:07:17 dist-test-slave-hhpq krb5kdc[7246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:07:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:07:17.121276 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:35177
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:35177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:17.232789 7259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:17.233124 7259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:17.233180 7259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:17.237318 7259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:17.237391 7259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:17.237416 7259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:17.237439 7259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:17.237468 7259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:17.242874 7259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:35177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:35177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:17.244027 7259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:17.245167 7259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:17.254398 7264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:17.254575 7259 server_base.cc:1034] running on GCE node
W20260927 15:07:17.254416 7265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:17.254410 7267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:17.255335 7259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:17.256541 7259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:17.257723 7259 hybrid_clock.cc:648] HybridClock initialized: now 1790521637257700 us; error 36 us; skew 500 ppm
Sep 27 15:07:17 dist-test-slave-hhpq krb5kdc[7246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521637, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:17.260581 7259 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:07:17.261556 7259 webserver.cc:460] Webserver started at http://127.7.17.254:38057/ using document root <none> and password file <none>
I20260927 15:07:17.262099 7259 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:17.262161 7259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:17.262368 7259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:17.264503 7259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "295fca53295c4add8cf5bcc16905fb62"
format_stamp: "Formatted at 2026-09-27 15:07:17 on dist-test-slave-hhpq"
server_key: "2d66286cb5cee990b847df9fa6777cd8"
server_key_iv: "99abea3b860dbcae36c06b5729387e2d"
server_key_version: "encryptionkey@0"
I20260927 15:07:17.265112 7259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "295fca53295c4add8cf5bcc16905fb62"
format_stamp: "Formatted at 2026-09-27 15:07:17 on dist-test-slave-hhpq"
server_key: "2d66286cb5cee990b847df9fa6777cd8"
server_key_iv: "99abea3b860dbcae36c06b5729387e2d"
server_key_version: "encryptionkey@0"
I20260927 15:07:17.269057 7259 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:07:17.271540 7274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:17.272794 7259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:07:17.272946 7259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "295fca53295c4add8cf5bcc16905fb62"
format_stamp: "Formatted at 2026-09-27 15:07:17 on dist-test-slave-hhpq"
server_key: "2d66286cb5cee990b847df9fa6777cd8"
server_key_iv: "99abea3b860dbcae36c06b5729387e2d"
server_key_version: "encryptionkey@0"
I20260927 15:07:17.273046 7259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:17.289541 7259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:17.292798 7259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:17.293022 7259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:17.300367 7259 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:35177
I20260927 15:07:17.300436 7326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:35177 every 8 connection(s)
I20260927 15:07:17.301436 7259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:17.304550 7327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:17.307981 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 7259
I20260927 15:07:17.308117 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:07:17.309577 7239 external_mini_cluster.cc:1428] Setting key 074c02469fe4c3ba926df5b58c5d56f2
I20260927 15:07:17.310806 7327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62: Bootstrap starting.
I20260927 15:07:17.313627 7327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:17.314525 7327 log.cc:824] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:17.316362 7327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62: No bootstrap required, opened a new log
I20260927 15:07:17.320272 7327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295fca53295c4add8cf5bcc16905fb62" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35177 } }
I20260927 15:07:17.320513 7327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:17.320562 7327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 295fca53295c4add8cf5bcc16905fb62, State: Initialized, Role: FOLLOWER
I20260927 15:07:17.321017 7327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295fca53295c4add8cf5bcc16905fb62" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35177 } }
I20260927 15:07:17.321152 7327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:17.321214 7327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:17.321308 7327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:17.322259 7327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295fca53295c4add8cf5bcc16905fb62" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35177 } }
I20260927 15:07:17.322585 7327 leader_election.cc:302] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 295fca53295c4add8cf5bcc16905fb62; no voters:
I20260927 15:07:17.322885 7327 leader_election.cc:288] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:17.322978 7332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:17.323181 7332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62 [term 1 LEADER]: Becoming Leader. State: Replica: 295fca53295c4add8cf5bcc16905fb62, State: Running, Role: LEADER
I20260927 15:07:17.323499 7332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295fca53295c4add8cf5bcc16905fb62" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35177 } }
I20260927 15:07:17.324239 7327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:17.324894 7334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 295fca53295c4add8cf5bcc16905fb62. Latest consensus state: current_term: 1 leader_uuid: "295fca53295c4add8cf5bcc16905fb62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295fca53295c4add8cf5bcc16905fb62" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35177 } } }
I20260927 15:07:17.325016 7334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:17.325713 7333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "295fca53295c4add8cf5bcc16905fb62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295fca53295c4add8cf5bcc16905fb62" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35177 } } }
I20260927 15:07:17.325878 7333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62 [sys.catalog]: This master's current role is: LEADER
Sep 27 15:07:17 dist-test-slave-hhpq krb5kdc[7246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
W20260927 15:07:17.329552 7347 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:07:17.329627 7347 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:07:17.329707 7339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:17.330468 7339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:17.335734 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:17.312869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49252 (local address 127.7.17.254:35177)
0927 15:07:17.313311 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:17.313327 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:17.314461 (+ 1134us) server_negotiation.cc:407] Connection header received
0927 15:07:17.317680 (+ 3219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:17.317698 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:17.318027 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:17.318334 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:17.320537 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:17.321717 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:17.323072 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:17.323712 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:17.323873 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:17.329013 (+ 5140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:17.329020 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:17.329037 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:17.329072 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:17.331294 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:17.332011 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:17.332019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:17.332026 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:17.332192 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:17.332626 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:17.332629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:17.332631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:17.333270 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:17.333591 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:17.334289 (+ 698us) server_negotiation.cc:299] Negotiation successful
0927 15:07:17.334638 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":130,"threads_started":1}
I20260927 15:07:17.337730 7339 catalog_manager.cc:1347] Generated new cluster ID: 6877c545c59e45d9b9158c3819e74a1a
I20260927 15:07:17.337797 7339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:17.363034 7339 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:17.364048 7339 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:17.373442 7339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 295fca53295c4add8cf5bcc16905fb62: Generated new TSK 0
I20260927 15:07:17.374198 7339 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
I20260927 15:07:17.407965 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:35177
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:17.541278 7353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:17.541563 7353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:17.541610 7353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:17.545686 7353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:17.545750 7353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:17.545816 7353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:07:17.551100 7353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:35177
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:17.552129 7353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:17.553135 7353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:17.562731 7359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:17.562736 7358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:17.562798 7353 server_base.cc:1034] running on GCE node
W20260927 15:07:17.562736 7361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:17.563460 7353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:17.564108 7353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:17.565264 7353 hybrid_clock.cc:648] HybridClock initialized: now 1790521637565245 us; error 32 us; skew 500 ppm
Sep 27 15:07:17 dist-test-slave-hhpq krb5kdc[7246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521637, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:17.568218 7353 init.cc:377] Logged in from keytab as kudu/127.7.17.193@KRBTEST.COM (short username kudu)
I20260927 15:07:17.569273 7353 webserver.cc:460] Webserver started at http://127.7.17.193:46617/ using document root <none> and password file <none>
I20260927 15:07:17.569875 7353 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:17.569947 7353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:17.570144 7353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:17.572307 7353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "1b7c8268fa4d4c71910387a0992eeb10"
format_stamp: "Formatted at 2026-09-27 15:07:17 on dist-test-slave-hhpq"
server_key: "9b995f1113bf9c60aacd2ab7ee03ff03"
server_key_iv: "5f32bf23e189ef1d7d68ebadf7e98639"
server_key_version: "encryptionkey@0"
I20260927 15:07:17.572968 7353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "1b7c8268fa4d4c71910387a0992eeb10"
format_stamp: "Formatted at 2026-09-27 15:07:17 on dist-test-slave-hhpq"
server_key: "9b995f1113bf9c60aacd2ab7ee03ff03"
server_key_iv: "5f32bf23e189ef1d7d68ebadf7e98639"
server_key_version: "encryptionkey@0"
I20260927 15:07:17.577142 7353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:07:17.579861 7368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:17.581282 7353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:17.581439 7353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "1b7c8268fa4d4c71910387a0992eeb10"
format_stamp: "Formatted at 2026-09-27 15:07:17 on dist-test-slave-hhpq"
server_key: "9b995f1113bf9c60aacd2ab7ee03ff03"
server_key_iv: "5f32bf23e189ef1d7d68ebadf7e98639"
server_key_version: "encryptionkey@0"
I20260927 15:07:17.581552 7353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:17.616268 7353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:17.619375 7353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:17.619578 7353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:17.620167 7353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:17.621026 7353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:17.621073 7353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:17.621127 7353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:17.621156 7353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:17.630551 7353 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:34081
I20260927 15:07:17.630586 7481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:34081 every 8 connection(s)
I20260927 15:07:17.631469 7353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:17.635577 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 7353
I20260927 15:07:17.635672 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:07:17.635974 7239 external_mini_cluster.cc:1428] Setting key b1b3753b3995b64a80e7009dc429d529
Sep 27 15:07:17 dist-test-slave-hhpq krb5kdc[7246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521637, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.17.194@KRBTEST.COM; defaulting to no policy
I20260927 15:07:17.646123 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:17.633395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:60155 (local address 127.7.17.254:35177)
0927 15:07:17.633531 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:17.633534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:17.634414 (+ 880us) server_negotiation.cc:407] Connection header received
0927 15:07:17.635774 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:17.635778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:17.635842 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:17.635950 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:17.637508 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:17.638241 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:17.639725 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:17.639977 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:17.640066 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:17.642528 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:17.642531 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:17.642533 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:17.642560 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:17.644042 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:17.644574 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:17.644578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:17.644580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:17.644656 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:17.645040 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:17.645043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:17.645045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:17.645226 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:17.645321 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:17.645829 (+ 508us) server_negotiation.cc:299] Negotiation successful
0927 15:07:17.645959 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:07:17.646943 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:17.633724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:35177 (local address 127.7.17.193:60155)
0927 15:07:17.634201 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:17.634249 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:17.635509 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:17.636126 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:17.636134 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:17.636590 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:17.637344 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:17.637365 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:17.638404 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:17.638407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:17.639590 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:17.639603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:17.640136 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:17.640140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:17.640316 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:17.640912 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:17.640930 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:17.642355 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:17.644170 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:17.644177 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:17.644192 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:17.644475 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:17.644738 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:17.644742 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:17.644744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:17.644954 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:17.645321 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:17.645328 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:17.645585 (+ 257us) client_negotiation.cc:769] Sending connection context
0927 15:07:17.645751 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 15:07:17.646100 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":125,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.194@KRBTEST.COM" created.
I20260927 15:07:17.648206 7482 heartbeater.cc:347] Connected to a master server at 127.7.17.254:35177
I20260927 15:07:17.648476 7482 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:17.649101 7482 heartbeater.cc:511] Master 127.7.17.254:35177 requested a full tablet report, sending...
I20260927 15:07:17.650830 7286 ts_manager.cc:194] Registered new tserver with Master: 1b7c8268fa4d4c71910387a0992eeb10 (127.7.17.193:34081)
I20260927 15:07:17.652450 7286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:60155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
I20260927 15:07:17.664296 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:35177
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:17.781776 7487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:17.782088 7487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:17.782135 7487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:17.786352 7487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:17.786438 7487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:17.786504 7487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:07:17.792059 7487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:35177
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:17.793217 7487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:17.794320 7487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:17.804329 7493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:17.804500 7487 server_base.cc:1034] running on GCE node
W20260927 15:07:17.804322 7492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:17.804370 7495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:17.805143 7487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:17.805824 7487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:17.806993 7487 hybrid_clock.cc:648] HybridClock initialized: now 1790521637806970 us; error 37 us; skew 500 ppm
Sep 27 15:07:17 dist-test-slave-hhpq krb5kdc[7246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521637, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:17.809939 7487 init.cc:377] Logged in from keytab as kudu/127.7.17.194@KRBTEST.COM (short username kudu)
I20260927 15:07:17.810909 7487 webserver.cc:460] Webserver started at http://127.7.17.194:44917/ using document root <none> and password file <none>
I20260927 15:07:17.811458 7487 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:17.811518 7487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:17.811694 7487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:17.813899 7487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "7c39ce5560c6454d8281acbe2812c87f"
format_stamp: "Formatted at 2026-09-27 15:07:17 on dist-test-slave-hhpq"
server_key: "c0efc632088340899d33a261ec77076b"
server_key_iv: "ccc87d97fc76369325c21c4b9626a797"
server_key_version: "encryptionkey@0"
I20260927 15:07:17.814497 7487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "7c39ce5560c6454d8281acbe2812c87f"
format_stamp: "Formatted at 2026-09-27 15:07:17 on dist-test-slave-hhpq"
server_key: "c0efc632088340899d33a261ec77076b"
server_key_iv: "ccc87d97fc76369325c21c4b9626a797"
server_key_version: "encryptionkey@0"
I20260927 15:07:17.818373 7487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:07:17.820969 7502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:17.822259 7487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 15:07:17.822372 7487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "7c39ce5560c6454d8281acbe2812c87f"
format_stamp: "Formatted at 2026-09-27 15:07:17 on dist-test-slave-hhpq"
server_key: "c0efc632088340899d33a261ec77076b"
server_key_iv: "ccc87d97fc76369325c21c4b9626a797"
server_key_version: "encryptionkey@0"
I20260927 15:07:17.822464 7487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:17.844002 7487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:17.847117 7487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:17.847281 7487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:17.847903 7487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:17.848742 7487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:17.848796 7487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:17.848861 7487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:17.848887 7487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:17.858390 7487 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:37957
I20260927 15:07:17.858414 7615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:37957 every 8 connection(s)
I20260927 15:07:17.859331 7487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:07:17.861002 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 7487
I20260927 15:07:17.861109 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:07:17.861353 7239 external_mini_cluster.cc:1428] Setting key eac5ec1822a96aa3b719884bc65d2d41
Sep 27 15:07:17 dist-test-slave-hhpq krb5kdc[7246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521637, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.195@KRBTEST.COM" created.
I20260927 15:07:17.873688 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:17.861225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:60327 (local address 127.7.17.254:35177)
0927 15:07:17.861385 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:17.861391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:17.862426 (+ 1035us) server_negotiation.cc:407] Connection header received
0927 15:07:17.863897 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:17.863901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:17.863954 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:17.864042 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:17.865345 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:17.865890 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:17.867059 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:17.867277 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:17.867368 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:17.870039 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:17.870043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:17.870045 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:17.870070 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:17.871432 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:17.872013 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:17.872017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:17.872018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:17.872097 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:17.872442 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:17.872445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:17.872447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:17.872677 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:17.872774 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:17.873373 (+ 599us) server_negotiation.cc:299] Negotiation successful
0927 15:07:17.873512 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:07:17.874686 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:17.861688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:35177 (local address 127.7.17.194:60327)
0927 15:07:17.862250 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:17.862313 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:17.863659 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:17.864258 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:17.864266 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:17.864643 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:17.865206 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:17.865222 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:17.866022 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:17.866026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:17.866936 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:17.866946 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:17.867419 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:17.867423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:17.867593 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:17.868143 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:17.868160 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:17.869834 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:17.871549 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:17.871555 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:17.871572 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:17.871910 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:17.872184 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:17.872187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:17.872189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:17.872352 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:17.872772 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:17.872779 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:17.873107 (+ 328us) client_negotiation.cc:769] Sending connection context
0927 15:07:17.873322 (+ 215us) client_negotiation.cc:240] Negotiation successful
0927 15:07:17.873733 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":149,"threads_started":1}
I20260927 15:07:17.875886 7616 heartbeater.cc:347] Connected to a master server at 127.7.17.254:35177
I20260927 15:07:17.876147 7616 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:17.876710 7616 heartbeater.cc:511] Master 127.7.17.254:35177 requested a full tablet report, sending...
I20260927 15:07:17.877823 7286 ts_manager.cc:194] Registered new tserver with Master: 7c39ce5560c6454d8281acbe2812c87f (127.7.17.194:37957)
I20260927 15:07:17.878603 7286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:60327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
I20260927 15:07:17.888556 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:35177
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:18.002182 7621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:18.002460 7621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:18.002507 7621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:18.006702 7621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:18.006776 7621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:18.006843 7621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:07:18.012480 7621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:35177
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:18.013670 7621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:18.014724 7621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:18.024690 7626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:18.024915 7629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:18.024725 7627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:18.025151 7621 server_base.cc:1034] running on GCE node
I20260927 15:07:18.025878 7621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:18.026604 7621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:18.027817 7621 hybrid_clock.cc:648] HybridClock initialized: now 1790521638027774 us; error 30 us; skew 500 ppm
Sep 27 15:07:18 dist-test-slave-hhpq krb5kdc[7246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521638, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:18.030830 7621 init.cc:377] Logged in from keytab as kudu/127.7.17.195@KRBTEST.COM (short username kudu)
I20260927 15:07:18.031798 7621 webserver.cc:460] Webserver started at http://127.7.17.195:44191/ using document root <none> and password file <none>
I20260927 15:07:18.032361 7621 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:18.032428 7621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:18.032632 7621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:18.034783 7621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "7a329defbdfa4828a004f0abaf030eb9"
format_stamp: "Formatted at 2026-09-27 15:07:18 on dist-test-slave-hhpq"
server_key: "90041ac21caaf2ea2e45670d1169520a"
server_key_iv: "8097629a12dc1ce5c053b86ea953d8f9"
server_key_version: "encryptionkey@0"
I20260927 15:07:18.035362 7621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "7a329defbdfa4828a004f0abaf030eb9"
format_stamp: "Formatted at 2026-09-27 15:07:18 on dist-test-slave-hhpq"
server_key: "90041ac21caaf2ea2e45670d1169520a"
server_key_iv: "8097629a12dc1ce5c053b86ea953d8f9"
server_key_version: "encryptionkey@0"
I20260927 15:07:18.039108 7621 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:07:18.041620 7636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:18.042958 7621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:18.043073 7621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "7a329defbdfa4828a004f0abaf030eb9"
format_stamp: "Formatted at 2026-09-27 15:07:18 on dist-test-slave-hhpq"
server_key: "90041ac21caaf2ea2e45670d1169520a"
server_key_iv: "8097629a12dc1ce5c053b86ea953d8f9"
server_key_version: "encryptionkey@0"
I20260927 15:07:18.043175 7621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:18.056931 7621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:18.059973 7621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:18.060164 7621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:18.060771 7621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:18.061686 7621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:18.061735 7621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:18.061791 7621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:18.061820 7621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:18.070950 7621 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:46723
I20260927 15:07:18.070971 7749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:46723 every 8 connection(s)
I20260927 15:07:18.071949 7621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:18.075017 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 7621
I20260927 15:07:18.075139 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:07:18.075438 7239 external_mini_cluster.cc:1428] Setting key ba2e30e83680d8c0046f4d273b437820
Sep 27 15:07:18 dist-test-slave-hhpq krb5kdc[7246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521638, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:18.086621 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.073859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:55295 (local address 127.7.17.254:35177)
0927 15:07:18.074018 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:18.074023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:18.074926 (+ 903us) server_negotiation.cc:407] Connection header received
0927 15:07:18.076572 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:18.076576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:18.076631 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:18.076712 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:18.078272 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.078881 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.080113 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.080339 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.080426 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:18.083058 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:18.083061 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:18.083063 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:18.083093 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:18.084379 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:18.084958 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:18.084961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:18.084963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:18.085042 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:18.085494 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:18.085499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:18.085501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:18.085718 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:18.085825 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:18.086335 (+ 510us) server_negotiation.cc:299] Negotiation successful
0927 15:07:18.086445 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:07:18.087517 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.074253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:35177 (local address 127.7.17.195:55295)
0927 15:07:18.074727 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:18.074785 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:18.076306 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:18.076982 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:18.076993 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:18.077405 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:18.078094 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:18.078117 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.079018 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.079021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:18.079993 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:18.080005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.080484 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.080487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:18.080644 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:18.081242 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:18.081259 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:18.082862 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:18.084515 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:18.084521 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:18.084542 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:18.084859 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:18.085118 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:18.085120 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:18.085122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:18.085266 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:18.085854 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:18.085859 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:18.086120 (+ 261us) client_negotiation.cc:769] Sending connection context
0927 15:07:18.086285 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 15:07:18.086616 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":133,"threads_started":1}
I20260927 15:07:18.088781 7750 heartbeater.cc:347] Connected to a master server at 127.7.17.254:35177
I20260927 15:07:18.089100 7750 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:18.089708 7750 heartbeater.cc:511] Master 127.7.17.254:35177 requested a full tablet report, sending...
I20260927 15:07:18.090746 7286 ts_manager.cc:194] Registered new tserver with Master: 7a329defbdfa4828a004f0abaf030eb9 (127.7.17.195:46723)
I20260927 15:07:18.091324 7286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:55295
I20260927 15:07:18.102340 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:07:18 dist-test-slave-hhpq krb5kdc[7246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521638, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:07:18 dist-test-slave-hhpq krb5kdc[7246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521638, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.17.193@KRBTEST.COM
I20260927 15:07:18.127941 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.118573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41702 (local address 127.7.17.193:34081)
0927 15:07:18.118862 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:18.118877 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:18.118906 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:07:18.118972 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:18.118979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:18.119209 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:18.119346 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:18.120082 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.120904 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.121723 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.122109 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.122203 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:18.124522 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:18.124530 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:18.124544 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:18.124568 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:18.126184 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:18.126608 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:18.126613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:18.126615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:18.126695 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:18.126946 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:18.126949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:18.126951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:18.127205 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:18.127420 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:18.127602 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 15:07:18.127758 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":63,"threads_started":1}
W20260927 15:07:18.128957 7396 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41702
Sep 27 15:07:18 dist-test-slave-hhpq krb5kdc[7246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:07:18 dist-test-slave-hhpq krb5kdc[7246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.193@KRBTEST.COM
I20260927 15:07:18.154891 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:18.146703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41706 (local address 127.7.17.193:34081)
0927 15:07:18.146869 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:18.146874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:18.146956 (+ 82us) server_negotiation.cc:407] Connection header received
0927 15:07:18.147127 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:18.147130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:18.147179 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:18.147249 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:18.148162 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.148679 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.149588 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:18.149832 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:18.149922 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:18.152169 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:18.152172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:18.152174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:18.152200 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:18.153401 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:18.153785 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:18.153790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:18.153791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:18.153864 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:18.154111 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:18.154115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:18.154116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:18.154281 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:18.154391 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:18.154616 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 15:07:18.154730 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:07:18.155905 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 7353
I20260927 15:07:18.160657 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 7487
I20260927 15:07:18.165429 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 7621
I20260927 15:07:18.170127 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 7259
2026-09-27T15:07:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2150 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:07:18 dist-test-slave-hhpq krb5kdc[7764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:07:18 dist-test-slave-hhpq krb5kdc[7764](info): set up 2 sockets
Sep 27 15:07:18 dist-test-slave-hhpq krb5kdc[7764](info): commencing operation
krb5kdc: starting...
W20260927 15:07:19.136397 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:07:19 dist-test-slave-hhpq krb5kdc[7764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:07:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:07:19.261098 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:46441
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:46441
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:19.372973 7777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:19.373293 7777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:19.373346 7777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:19.377490 7777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:19.377550 7777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:19.377569 7777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:19.377585 7777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:19.377601 7777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:19.383028 7777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:46441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:46441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:19.384153 7777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:19.385303 7777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:19.393929 7785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:19.393950 7777 server_base.cc:1034] running on GCE node
W20260927 15:07:19.393971 7783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:19.393941 7782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:19.394961 7777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:19.396206 7777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:19.397367 7777 hybrid_clock.cc:648] HybridClock initialized: now 1790521639397344 us; error 41 us; skew 500 ppm
Sep 27 15:07:19 dist-test-slave-hhpq krb5kdc[7764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521639, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:19.400163 7777 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:07:19.401125 7777 webserver.cc:460] Webserver started at http://127.7.17.254:39415/ using document root <none> and password file <none>
I20260927 15:07:19.401659 7777 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:19.401717 7777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:19.401922 7777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:19.404055 7777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "648ed47f29be49cd90257f5dc24f827f"
format_stamp: "Formatted at 2026-09-27 15:07:19 on dist-test-slave-hhpq"
server_key: "7881fad3b8e207dad5bede6dd1afb9aa"
server_key_iv: "e486d9f851bbff480d4eba70a1142c23"
server_key_version: "encryptionkey@0"
I20260927 15:07:19.404647 7777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "648ed47f29be49cd90257f5dc24f827f"
format_stamp: "Formatted at 2026-09-27 15:07:19 on dist-test-slave-hhpq"
server_key: "7881fad3b8e207dad5bede6dd1afb9aa"
server_key_iv: "e486d9f851bbff480d4eba70a1142c23"
server_key_version: "encryptionkey@0"
I20260927 15:07:19.408677 7777 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:07:19.411329 7792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:19.412681 7777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:07:19.412880 7777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "648ed47f29be49cd90257f5dc24f827f"
format_stamp: "Formatted at 2026-09-27 15:07:19 on dist-test-slave-hhpq"
server_key: "7881fad3b8e207dad5bede6dd1afb9aa"
server_key_iv: "e486d9f851bbff480d4eba70a1142c23"
server_key_version: "encryptionkey@0"
I20260927 15:07:19.413013 7777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:19.427856 7777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:19.431159 7777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:19.431354 7777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:19.438267 7777 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:46441
I20260927 15:07:19.438288 7844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:46441 every 8 connection(s)
I20260927 15:07:19.439306 7777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:19.442318 7845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:19.447523 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 7777
I20260927 15:07:19.447654 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:07:19.447919 7239 external_mini_cluster.cc:1428] Setting key 52abd0f992c82df0ff94f447fb859380
I20260927 15:07:19.448661 7845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f: Bootstrap starting.
I20260927 15:07:19.451171 7845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:19.451917 7845 log.cc:824] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:19.454005 7845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f: No bootstrap required, opened a new log
Sep 27 15:07:19 dist-test-slave-hhpq krb5kdc[7764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:19.458359 7845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "648ed47f29be49cd90257f5dc24f827f" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46441 } }
I20260927 15:07:19.458616 7845 raft_consensus.cc:374] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:19.458664 7845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648ed47f29be49cd90257f5dc24f827f, State: Initialized, Role: FOLLOWER
I20260927 15:07:19.459185 7845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "648ed47f29be49cd90257f5dc24f827f" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46441 } }
I20260927 15:07:19.459342 7845 raft_consensus.cc:388] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:19.459411 7845 raft_consensus.cc:482] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:19.459511 7845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:19.460758 7845 raft_consensus.cc:504] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "648ed47f29be49cd90257f5dc24f827f" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46441 } }
I20260927 15:07:19.461158 7845 leader_election.cc:302] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 648ed47f29be49cd90257f5dc24f827f; no voters:
I20260927 15:07:19.461508 7845 leader_election.cc:288] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:19.461599 7850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:19.461826 7850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f [term 1 LEADER]: Becoming Leader. State: Replica: 648ed47f29be49cd90257f5dc24f827f, State: Running, Role: LEADER
I20260927 15:07:19.462208 7850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "648ed47f29be49cd90257f5dc24f827f" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46441 } }
I20260927 15:07:19.462984 7845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:19.463174 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:19.449570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54618 (local address 127.7.17.254:46441)
0927 15:07:19.450009 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:19.450024 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:19.450059 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:07:19.450718 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:19.450736 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:19.451093 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:19.451435 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:19.452264 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:19.453328 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:19.454169 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:19.454864 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:19.455042 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:19.457757 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:19.457763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:19.457780 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:19.457811 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:19.459795 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:19.460338 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:19.460367 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:19.460373 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:19.460505 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:19.460898 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:19.460901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:19.460902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:19.461434 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:19.461624 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:19.461917 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 15:07:19.462262 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":129,"threads_started":1}
I20260927 15:07:19.463977 7852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 648ed47f29be49cd90257f5dc24f827f. Latest consensus state: current_term: 1 leader_uuid: "648ed47f29be49cd90257f5dc24f827f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "648ed47f29be49cd90257f5dc24f827f" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46441 } } }
I20260927 15:07:19.464102 7852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:19.465476 7851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "648ed47f29be49cd90257f5dc24f827f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "648ed47f29be49cd90257f5dc24f827f" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46441 } } }
I20260927 15:07:19.465579 7851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f [sys.catalog]: This master's current role is: LEADER
W20260927 15:07:19.468034 7865 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:07:19.468115 7865 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:07:19.468206 7859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:19.469226 7859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:19.475072 7859 catalog_manager.cc:1347] Generated new cluster ID: d72afd759e0940e19b9660a291b982d5
I20260927 15:07:19.475157 7859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:19.486209 7859 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:19.487255 7859 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:19.497633 7859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 648ed47f29be49cd90257f5dc24f827f: Generated new TSK 0
I20260927 15:07:19.498539 7859 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
I20260927 15:07:19.536094 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46441
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:19.675861 7871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:19.676168 7871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:19.676230 7871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:19.680419 7871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:19.680507 7871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:19.680594 7871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:07:19.686388 7871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46441
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:19.687588 7871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:19.688663 7871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:19.698983 7879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:19.698961 7877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:19.699081 7871 server_base.cc:1034] running on GCE node
W20260927 15:07:19.698961 7876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:19.699677 7871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:19.700351 7871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:19.701516 7871 hybrid_clock.cc:648] HybridClock initialized: now 1790521639701486 us; error 44 us; skew 500 ppm
Sep 27 15:07:19 dist-test-slave-hhpq krb5kdc[7764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521639, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:19.704241 7871 init.cc:377] Logged in from keytab as kudu/127.7.17.193@KRBTEST.COM (short username kudu)
I20260927 15:07:19.705192 7871 webserver.cc:460] Webserver started at http://127.7.17.193:37329/ using document root <none> and password file <none>
I20260927 15:07:19.705751 7871 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:19.705817 7871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:19.706014 7871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:19.708087 7871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "303977a3c371419fb23c244c6ee590c5"
format_stamp: "Formatted at 2026-09-27 15:07:19 on dist-test-slave-hhpq"
server_key: "0cebb1d2492d5b5ef720b68a764fdb7f"
server_key_iv: "6e5a9c3466e37a803498072c08b5f75a"
server_key_version: "encryptionkey@0"
I20260927 15:07:19.708630 7871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "303977a3c371419fb23c244c6ee590c5"
format_stamp: "Formatted at 2026-09-27 15:07:19 on dist-test-slave-hhpq"
server_key: "0cebb1d2492d5b5ef720b68a764fdb7f"
server_key_iv: "6e5a9c3466e37a803498072c08b5f75a"
server_key_version: "encryptionkey@0"
I20260927 15:07:19.712455 7871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:07:19.715097 7886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:19.716387 7871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:07:19.716506 7871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "303977a3c371419fb23c244c6ee590c5"
format_stamp: "Formatted at 2026-09-27 15:07:19 on dist-test-slave-hhpq"
server_key: "0cebb1d2492d5b5ef720b68a764fdb7f"
server_key_iv: "6e5a9c3466e37a803498072c08b5f75a"
server_key_version: "encryptionkey@0"
I20260927 15:07:19.716620 7871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:19.730578 7871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:19.734035 7871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:19.734225 7871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:19.734866 7871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:19.735821 7871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:19.735883 7871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:19.735942 7871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:19.735971 7871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:19.744867 7871 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:41063
I20260927 15:07:19.744896 7999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:41063 every 8 connection(s)
I20260927 15:07:19.745820 7871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:19.753423 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 7871
I20260927 15:07:19.753545 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:07:19.753819 7239 external_mini_cluster.cc:1428] Setting key 26c19bf863077174dd0a9ca05c65f155
Sep 27 15:07:19 dist-test-slave-hhpq krb5kdc[7764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521639, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:19.759761 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:19.747723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:56929 (local address 127.7.17.254:46441)
0927 15:07:19.747845 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:19.747849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:19.748735 (+ 886us) server_negotiation.cc:407] Connection header received
0927 15:07:19.750124 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:19.750128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:19.750177 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:19.750251 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:19.751387 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:19.751951 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:19.753117 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:19.753373 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:19.753461 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:19.755907 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:19.755911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:19.755914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:19.755944 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:19.757512 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:19.758106 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:19.758109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:19.758110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:19.758187 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:19.758558 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:19.758561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:19.758562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:19.758751 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:19.758826 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:19.759525 (+ 699us) server_negotiation.cc:299] Negotiation successful
0927 15:07:19.759634 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:07:19.761199 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:19.748106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46441 (local address 127.7.17.193:56929)
0927 15:07:19.748565 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:19.748609 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:19.749917 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:19.750398 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:19.750405 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:19.750760 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:19.751248 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:19.751262 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:19.752045 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:19.752048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:19.753011 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:19.753020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:19.753533 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:19.753537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:19.753740 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:19.754339 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:19.754356 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:19.755741 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:19.757630 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:19.757637 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:19.757655 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:19.757999 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:19.758263 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:19.758266 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:19.758268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:19.758461 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:19.758850 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:19.758857 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:19.759223 (+ 366us) client_negotiation.cc:769] Sending connection context
0927 15:07:19.759462 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 15:07:19.759952 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":124,"threads_started":1}
I20260927 15:07:19.762763 8000 heartbeater.cc:347] Connected to a master server at 127.7.17.254:46441
I20260927 15:07:19.763059 8000 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.7.17.194@KRBTEST.COM; defaulting to no policy
I20260927 15:07:19.763752 8000 heartbeater.cc:511] Master 127.7.17.254:46441 requested a full tablet report, sending...
I20260927 15:07:19.765743 7809 ts_manager.cc:194] Registered new tserver with Master: 303977a3c371419fb23c244c6ee590c5 (127.7.17.193:41063)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.194@KRBTEST.COM" created.
I20260927 15:07:19.767201 7809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:56929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
I20260927 15:07:19.783226 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46441
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:19.897944 8005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:19.898240 8005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:19.898306 8005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:19.902609 8005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:19.902694 8005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:19.902778 8005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:07:19.908340 8005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46441
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:19.909588 8005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:19.910627 8005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:19.920922 8011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:19.920922 8010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:19.921248 8005 server_base.cc:1034] running on GCE node
W20260927 15:07:19.921522 8013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:19.921953 8005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:19.922703 8005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:19.923862 8005 hybrid_clock.cc:648] HybridClock initialized: now 1790521639923836 us; error 44 us; skew 500 ppm
Sep 27 15:07:19 dist-test-slave-hhpq krb5kdc[7764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521639, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:19.926873 8005 init.cc:377] Logged in from keytab as kudu/127.7.17.194@KRBTEST.COM (short username kudu)
I20260927 15:07:19.927803 8005 webserver.cc:460] Webserver started at http://127.7.17.194:38753/ using document root <none> and password file <none>
I20260927 15:07:19.928365 8005 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:19.928427 8005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:19.928593 8005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:19.930863 8005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "4888621aa8c04e24b888bdf94bb5db9d"
format_stamp: "Formatted at 2026-09-27 15:07:19 on dist-test-slave-hhpq"
server_key: "0ed1842fe97e320de8c67ee882f28f00"
server_key_iv: "ae86b974f5bd728eded1cbccd674c63e"
server_key_version: "encryptionkey@0"
I20260927 15:07:19.931412 8005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "4888621aa8c04e24b888bdf94bb5db9d"
format_stamp: "Formatted at 2026-09-27 15:07:19 on dist-test-slave-hhpq"
server_key: "0ed1842fe97e320de8c67ee882f28f00"
server_key_iv: "ae86b974f5bd728eded1cbccd674c63e"
server_key_version: "encryptionkey@0"
I20260927 15:07:19.935403 8005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:07:19.938213 8020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:19.939726 8005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:07:19.939855 8005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "4888621aa8c04e24b888bdf94bb5db9d"
format_stamp: "Formatted at 2026-09-27 15:07:19 on dist-test-slave-hhpq"
server_key: "0ed1842fe97e320de8c67ee882f28f00"
server_key_iv: "ae86b974f5bd728eded1cbccd674c63e"
server_key_version: "encryptionkey@0"
I20260927 15:07:19.939989 8005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:19.954571 8005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:19.957839 8005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:19.958029 8005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:19.958636 8005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:19.959501 8005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:19.959547 8005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:19.959590 8005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:19.959617 8005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:19.968879 8005 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:39529
I20260927 15:07:19.968920 8133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:39529 every 8 connection(s)
I20260927 15:07:19.969918 8005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
Sep 27 15:07:19 dist-test-slave-hhpq krb5kdc[7764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521639, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:19.979878 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 8005
I20260927 15:07:19.980005 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:07:19.980244 7239 external_mini_cluster.cc:1428] Setting key 24fbae05c3541827c2ec54c2a8d8a52a
I20260927 15:07:19.984086 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:19.971795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:37307 (local address 127.7.17.254:46441)
0927 15:07:19.971928 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:19.971932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:19.972899 (+ 967us) server_negotiation.cc:407] Connection header received
0927 15:07:19.974260 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:19.974263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:19.974315 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:19.974393 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:19.975705 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:19.976243 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:19.977513 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:19.977734 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:19.977817 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:19.980469 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:19.980473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:19.980476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:19.980504 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:19.982009 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:19.982525 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:19.982529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:19.982531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:19.982625 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:19.982951 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:19.982954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:19.982956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:19.983174 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:19.983259 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:19.983768 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 15:07:19.983915 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:07:19.985023 8136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:19.972244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46441 (local address 127.7.17.194:37307)
0927 15:07:19.972706 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:19.972752 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:19.974029 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:19.974626 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:19.974635 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:19.975032 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:19.975548 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:19.975562 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:19.976354 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:19.976356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:19.977372 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:19.977381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:19.977894 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:19.977897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:19.978050 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:19.978631 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:19.978648 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:19.980270 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:19.982124 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:19.982129 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:19.982144 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:19.982425 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:19.982709 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:19.982711 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:19.982712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:19.982873 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:19.983261 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:19.983266 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:19.983522 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 15:07:19.983688 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 15:07:19.983977 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":127,"threads_started":1}
I20260927 15:07:19.986291 8134 heartbeater.cc:347] Connected to a master server at 127.7.17.254:46441
I20260927 15:07:19.986542 8134 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:19.987030 8134 heartbeater.cc:511] Master 127.7.17.254:46441 requested a full tablet report, sending...
I20260927 15:07:19.988089 7809 ts_manager.cc:194] Registered new tserver with Master: 4888621aa8c04e24b888bdf94bb5db9d (127.7.17.194:39529)
I20260927 15:07:19.988667 7809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:37307
WARNING: no policy specified for kudu/127.7.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
I20260927 15:07:20.010493 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46441
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:20.121990 8139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:20.122260 8139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:20.122304 8139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:20.126330 8139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:20.126396 8139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:20.126457 8139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:07:20.131866 8139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46441
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:20.132985 8139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:20.134012 8139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:20.143683 8147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:20.143726 8144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:20.143775 8139 server_base.cc:1034] running on GCE node
W20260927 15:07:20.143724 8145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:20.144266 8139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:20.144973 8139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:20.146150 8139 hybrid_clock.cc:648] HybridClock initialized: now 1790521640146133 us; error 25 us; skew 500 ppm
Sep 27 15:07:20 dist-test-slave-hhpq krb5kdc[7764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521640, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:20.148814 8139 init.cc:377] Logged in from keytab as kudu/127.7.17.195@KRBTEST.COM (short username kudu)
I20260927 15:07:20.149837 8139 webserver.cc:460] Webserver started at http://127.7.17.195:41643/ using document root <none> and password file <none>
I20260927 15:07:20.150488 8139 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:20.150558 8139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:20.150792 8139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:20.153378 8139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "5cb8b28e8a444ffbbad0c387993b894a"
format_stamp: "Formatted at 2026-09-27 15:07:20 on dist-test-slave-hhpq"
server_key: "b22112c97e19e634fc40c3725f9b039d"
server_key_iv: "a75d72bd4f7d4ec342701072645b796f"
server_key_version: "encryptionkey@0"
I20260927 15:07:20.153928 8139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "5cb8b28e8a444ffbbad0c387993b894a"
format_stamp: "Formatted at 2026-09-27 15:07:20 on dist-test-slave-hhpq"
server_key: "b22112c97e19e634fc40c3725f9b039d"
server_key_iv: "a75d72bd4f7d4ec342701072645b796f"
server_key_version: "encryptionkey@0"
I20260927 15:07:20.157703 8139 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:07:20.160341 8154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:20.161772 8139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:20.161900 8139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "5cb8b28e8a444ffbbad0c387993b894a"
format_stamp: "Formatted at 2026-09-27 15:07:20 on dist-test-slave-hhpq"
server_key: "b22112c97e19e634fc40c3725f9b039d"
server_key_iv: "a75d72bd4f7d4ec342701072645b796f"
server_key_version: "encryptionkey@0"
I20260927 15:07:20.162021 8139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:20.181806 8139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:20.185295 8139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:20.185570 8139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:20.186203 8139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:20.187074 8139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:20.187124 8139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:20.187177 8139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:20.187206 8139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:20.195948 8139 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:45787
I20260927 15:07:20.196008 8267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:45787 every 8 connection(s)
I20260927 15:07:20.196985 8139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
Sep 27 15:07:20 dist-test-slave-hhpq krb5kdc[7764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521640, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:20.207064 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 8139
I20260927 15:07:20.207186 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:07:20.207489 7239 external_mini_cluster.cc:1428] Setting key 980b38e35433cc1ed66ae95875b129b7
I20260927 15:07:20.210953 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:20.198784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:50941 (local address 127.7.17.254:46441)
0927 15:07:20.198929 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:20.198933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:20.199808 (+ 875us) server_negotiation.cc:407] Connection header received
0927 15:07:20.201231 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:20.201235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:20.201295 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:20.201373 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:20.202649 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.203177 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.204330 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.204556 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.204646 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:20.207427 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:20.207430 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:20.207432 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:20.207460 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:20.208969 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:20.209594 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:20.209596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:20.209598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:20.209672 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:20.209955 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:20.209957 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:20.209958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:20.210140 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:20.210213 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:20.210717 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 15:07:20.210820 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:07:20.211798 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:20.199199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46441 (local address 127.7.17.195:50941)
0927 15:07:20.199650 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:20.199696 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:20.201012 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:20.201571 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:20.201579 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:20.201985 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:20.202499 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:20.202514 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.203297 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.203299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:20.204211 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:20.204220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.204700 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.204704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:20.204913 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:20.205479 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:20.205495 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:20.207232 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:20.209078 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:20.209085 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:20.209103 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:20.209504 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:20.209734 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:20.209736 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:20.209738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:20.209884 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:20.210213 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:20.210218 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:20.210518 (+ 300us) client_negotiation.cc:769] Sending connection context
0927 15:07:20.210679 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 15:07:20.210947 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":121,"threads_started":1}
I20260927 15:07:20.213021 8268 heartbeater.cc:347] Connected to a master server at 127.7.17.254:46441
I20260927 15:07:20.213265 8268 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:20.213784 8268 heartbeater.cc:511] Master 127.7.17.254:46441 requested a full tablet report, sending...
I20260927 15:07:20.214963 7809 ts_manager.cc:194] Registered new tserver with Master: 5cb8b28e8a444ffbbad0c387993b894a (127.7.17.195:45787)
I20260927 15:07:20.215593 7809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:50941
I20260927 15:07:20.221873 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:07:20.233035 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:20.225539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54630 (local address 127.7.17.254:46441)
0927 15:07:20.225684 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:20.225688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:20.225855 (+ 167us) server_negotiation.cc:407] Connection header received
0927 15:07:20.225957 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:20.225959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:20.226008 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:20.226072 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:20.226861 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.227427 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.228291 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.228537 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.228622 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:20.230055 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:20.230058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:20.230060 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:20.230087 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:20.231429 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:20.231871 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:20.231874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:20.231875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:20.231967 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:20.232227 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:20.232229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:20.232230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:20.232456 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:20.232557 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:20.232770 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 15:07:20.232911 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:07:20.238008 7809 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:07:20.240970 7809 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:20.255535 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:20.250253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:39529 (local address 127.0.0.1:42170)
0927 15:07:20.250691 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:20.250702 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:20.250772 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:20.251511 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:20.251517 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:20.251538 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:20.251690 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:20.251697 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.252782 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.252785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:20.254049 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:20.254059 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.254998 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.255002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:20.255105 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:20.255138 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:07:20.255244 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 15:07:20.255381 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":107,"threads_started":1}
I20260927 15:07:20.255539 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:20.250643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42170 (local address 127.7.17.194:39529)
0927 15:07:20.251063 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:20.251075 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:20.251097 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:07:20.251167 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:20.251171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:20.251358 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:20.251508 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:20.251808 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.252654 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.254185 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.254891 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.254986 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:20.255137 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:20.255285 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 15:07:20.255380 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":111,"threads_started":1}
I20260927 15:07:20.255999 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:20.249854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:45787 (local address 127.0.0.1:36240)
0927 15:07:20.250571 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:20.250625 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:20.250782 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:20.251707 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:20.251709 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:20.251726 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:20.251857 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:20.251863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.253265 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.253268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:20.254448 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:20.254457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.255562 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.255565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:20.255696 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:20.255717 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:07:20.255790 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 15:07:20.255857 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":101,"spinlock_wait_cycles":13056,"thread_start_us":117,"threads_started":1}
I20260927 15:07:20.256304 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:20.249912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54488 (local address 127.7.17.193:41063)
0927 15:07:20.251190 (+ 1278us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:20.251202 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:20.251220 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:07:20.251292 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:20.251296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:20.251454 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:20.251572 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:20.251989 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.253143 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.254405 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.255231 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.255344 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:20.255488 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:20.256071 (+ 583us) server_negotiation.cc:299] Negotiation successful
0927 15:07:20.256165 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1190,"thread_start_us":54,"threads_started":1}
I20260927 15:07:20.256348 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:20.249767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:41063 (local address 127.0.0.1:54488)
0927 15:07:20.250571 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:20.250625 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:20.250770 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:20.251683 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:20.251686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:20.251703 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:20.251841 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:20.251849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.253293 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.253296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:20.254278 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:20.254288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.255550 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.255554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:20.255937 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:20.255957 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 15:07:20.256029 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 15:07:20.256216 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":273,"spinlock_wait_cycles":17024,"thread_start_us":81,"threads_started":1}
I20260927 15:07:20.256855 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:20.250172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36240 (local address 127.7.17.195:45787)
0927 15:07:20.250574 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:20.250585 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:20.250907 (+ 322us) server_negotiation.cc:407] Connection header received
0927 15:07:20.250976 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:20.250980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:20.251178 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:20.251738 (+ 560us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:20.251939 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.253127 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.254570 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.255329 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.255411 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:20.255516 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:20.255838 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 15:07:20.256673 (+ 835us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":135,"threads_started":1}
I20260927 15:07:20.258790 8068 tablet_service.cc:1467] Processing CreateTablet for tablet 43f54a8920f6409e88b3a4d0ec3812af (DEFAULT_TABLE table=test-table [id=6e62361764e549ec8fe72a08112dfeb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:20.259195 8202 tablet_service.cc:1467] Processing CreateTablet for tablet 43f54a8920f6409e88b3a4d0ec3812af (DEFAULT_TABLE table=test-table [id=6e62361764e549ec8fe72a08112dfeb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:20.259205 7934 tablet_service.cc:1467] Processing CreateTablet for tablet 43f54a8920f6409e88b3a4d0ec3812af (DEFAULT_TABLE table=test-table [id=6e62361764e549ec8fe72a08112dfeb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:20.259941 8068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f54a8920f6409e88b3a4d0ec3812af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:20.260260 8202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f54a8920f6409e88b3a4d0ec3812af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:20.260260 7934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f54a8920f6409e88b3a4d0ec3812af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:20.267448 8287 tablet_bootstrap.cc:492] T 43f54a8920f6409e88b3a4d0ec3812af P 5cb8b28e8a444ffbbad0c387993b894a: Bootstrap starting.
I20260927 15:07:20.268481 8285 tablet_bootstrap.cc:492] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d: Bootstrap starting.
I20260927 15:07:20.269097 8286 tablet_bootstrap.cc:492] T 43f54a8920f6409e88b3a4d0ec3812af P 303977a3c371419fb23c244c6ee590c5: Bootstrap starting.
I20260927 15:07:20.269973 8287 tablet_bootstrap.cc:654] T 43f54a8920f6409e88b3a4d0ec3812af P 5cb8b28e8a444ffbbad0c387993b894a: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:20.270823 8286 tablet_bootstrap.cc:654] T 43f54a8920f6409e88b3a4d0ec3812af P 303977a3c371419fb23c244c6ee590c5: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:20.271103 8285 tablet_bootstrap.cc:654] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:20.271315 8287 log.cc:824] T 43f54a8920f6409e88b3a4d0ec3812af P 5cb8b28e8a444ffbbad0c387993b894a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:20.271535 8286 log.cc:824] T 43f54a8920f6409e88b3a4d0ec3812af P 303977a3c371419fb23c244c6ee590c5: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:20.271852 8285 log.cc:824] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:20.273479 8287 tablet_bootstrap.cc:492] T 43f54a8920f6409e88b3a4d0ec3812af P 5cb8b28e8a444ffbbad0c387993b894a: No bootstrap required, opened a new log
I20260927 15:07:20.273646 8286 tablet_bootstrap.cc:492] T 43f54a8920f6409e88b3a4d0ec3812af P 303977a3c371419fb23c244c6ee590c5: No bootstrap required, opened a new log
I20260927 15:07:20.273680 8287 ts_tablet_manager.cc:1397] T 43f54a8920f6409e88b3a4d0ec3812af P 5cb8b28e8a444ffbbad0c387993b894a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:07:20.273800 8286 ts_tablet_manager.cc:1397] T 43f54a8920f6409e88b3a4d0ec3812af P 303977a3c371419fb23c244c6ee590c5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 15:07:20.273931 8285 tablet_bootstrap.cc:492] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d: No bootstrap required, opened a new log
I20260927 15:07:20.274111 8285 ts_tablet_manager.cc:1397] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:07:20.276988 8285 raft_consensus.cc:348] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303977a3c371419fb23c244c6ee590c5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41063 } } peers { permanent_uuid: "5cb8b28e8a444ffbbad0c387993b894a" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 45787 } } peers { permanent_uuid: "4888621aa8c04e24b888bdf94bb5db9d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39529 } }
I20260927 15:07:20.277242 8285 raft_consensus.cc:374] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:20.277410 8285 raft_consensus.cc:729] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4888621aa8c04e24b888bdf94bb5db9d, State: Initialized, Role: FOLLOWER
I20260927 15:07:20.277572 8287 raft_consensus.cc:348] T 43f54a8920f6409e88b3a4d0ec3812af P 5cb8b28e8a444ffbbad0c387993b894a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303977a3c371419fb23c244c6ee590c5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41063 } } peers { permanent_uuid: "5cb8b28e8a444ffbbad0c387993b894a" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 45787 } } peers { permanent_uuid: "4888621aa8c04e24b888bdf94bb5db9d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39529 } }
I20260927 15:07:20.277861 8287 raft_consensus.cc:374] T 43f54a8920f6409e88b3a4d0ec3812af P 5cb8b28e8a444ffbbad0c387993b894a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:20.277969 8287 raft_consensus.cc:729] T 43f54a8920f6409e88b3a4d0ec3812af P 5cb8b28e8a444ffbbad0c387993b894a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb8b28e8a444ffbbad0c387993b894a, State: Initialized, Role: FOLLOWER
I20260927 15:07:20.277918 8285 consensus_queue.cc:260] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303977a3c371419fb23c244c6ee590c5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41063 } } peers { permanent_uuid: "5cb8b28e8a444ffbbad0c387993b894a" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 45787 } } peers { permanent_uuid: "4888621aa8c04e24b888bdf94bb5db9d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39529 } }
I20260927 15:07:20.278033 8286 raft_consensus.cc:348] T 43f54a8920f6409e88b3a4d0ec3812af P 303977a3c371419fb23c244c6ee590c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303977a3c371419fb23c244c6ee590c5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41063 } } peers { permanent_uuid: "5cb8b28e8a444ffbbad0c387993b894a" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 45787 } } peers { permanent_uuid: "4888621aa8c04e24b888bdf94bb5db9d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39529 } }
I20260927 15:07:20.278301 8286 raft_consensus.cc:374] T 43f54a8920f6409e88b3a4d0ec3812af P 303977a3c371419fb23c244c6ee590c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:20.278358 8286 raft_consensus.cc:729] T 43f54a8920f6409e88b3a4d0ec3812af P 303977a3c371419fb23c244c6ee590c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 303977a3c371419fb23c244c6ee590c5, State: Initialized, Role: FOLLOWER
I20260927 15:07:20.278447 8287 consensus_queue.cc:260] T 43f54a8920f6409e88b3a4d0ec3812af P 5cb8b28e8a444ffbbad0c387993b894a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303977a3c371419fb23c244c6ee590c5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41063 } } peers { permanent_uuid: "5cb8b28e8a444ffbbad0c387993b894a" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 45787 } } peers { permanent_uuid: "4888621aa8c04e24b888bdf94bb5db9d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39529 } }
I20260927 15:07:20.278739 8286 consensus_queue.cc:260] T 43f54a8920f6409e88b3a4d0ec3812af P 303977a3c371419fb23c244c6ee590c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303977a3c371419fb23c244c6ee590c5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41063 } } peers { permanent_uuid: "5cb8b28e8a444ffbbad0c387993b894a" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 45787 } } peers { permanent_uuid: "4888621aa8c04e24b888bdf94bb5db9d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39529 } }
I20260927 15:07:20.278957 8285 ts_tablet_manager.cc:1428] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 15:07:20.279235 8268 heartbeater.cc:503] Master 127.7.17.254:46441 was elected leader, sending a full tablet report...
I20260927 15:07:20.279570 8287 ts_tablet_manager.cc:1428] T 43f54a8920f6409e88b3a4d0ec3812af P 5cb8b28e8a444ffbbad0c387993b894a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:07:20.279691 8286 ts_tablet_manager.cc:1428] T 43f54a8920f6409e88b3a4d0ec3812af P 303977a3c371419fb23c244c6ee590c5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:07:20.280009 8000 heartbeater.cc:503] Master 127.7.17.254:46441 was elected leader, sending a full tablet report...
I20260927 15:07:20.280670 8134 heartbeater.cc:503] Master 127.7.17.254:46441 was elected leader, sending a full tablet report...
I20260927 15:07:20.339226 8291 raft_consensus.cc:482] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:20.339504 8291 raft_consensus.cc:504] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303977a3c371419fb23c244c6ee590c5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41063 } } peers { permanent_uuid: "5cb8b28e8a444ffbbad0c387993b894a" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 45787 } } peers { permanent_uuid: "4888621aa8c04e24b888bdf94bb5db9d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39529 } }
I20260927 15:07:20.340723 8291 leader_election.cc:288] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 303977a3c371419fb23c244c6ee590c5 (127.7.17.193:41063), 5cb8b28e8a444ffbbad0c387993b894a (127.7.17.195:45787)
I20260927 15:07:20.344985 8136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:20.341044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:45787 (local address 127.7.17.194:54397)
0927 15:07:20.341163 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:20.341179 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:20.341326 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:20.341668 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:20.341672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:20.341691 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:20.341796 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:20.341802 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.342683 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.342686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:20.343756 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:20.343766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.344505 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.344508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:20.344607 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:20.344631 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:07:20.344698 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 15:07:20.344774 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":45}
I20260927 15:07:20.345105 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:20.341134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:52259 (local address 127.7.17.193:41063)
0927 15:07:20.341259 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:20.341262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:20.341854 (+ 592us) server_negotiation.cc:407] Connection header received
0927 15:07:20.341915 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:20.341918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:20.341984 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:20.342039 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:20.342297 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.342908 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.344062 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.344536 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.344664 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:20.344711 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:20.344926 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 15:07:20.344999 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:07:20.345114 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:20.341134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:54397 (local address 127.7.17.195:45787)
0927 15:07:20.341258 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:20.341262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:20.341280 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:07:20.341484 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:20.341488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:20.341532 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:20.341630 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:20.341956 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.342574 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.343871 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.344314 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.344504 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:20.344549 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:20.344924 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 15:07:20.344996 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:07:20.345155 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:20.341044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:41063 (local address 127.7.17.194:52259)
0927 15:07:20.341748 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:20.341759 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:20.341854 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:20.342070 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:20.342073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:20.342089 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:20.342203 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:20.342209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.343034 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.343037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:20.343945 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:20.343956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.344646 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.344649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:20.344749 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:20.344773 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:07:20.344904 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 15:07:20.344972 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":359,"thread_start_us":117,"threads_started":1}
I20260927 15:07:20.345731 8222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f54a8920f6409e88b3a4d0ec3812af" candidate_uuid: "4888621aa8c04e24b888bdf94bb5db9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb8b28e8a444ffbbad0c387993b894a" is_pre_election: true
I20260927 15:07:20.345763 7954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f54a8920f6409e88b3a4d0ec3812af" candidate_uuid: "4888621aa8c04e24b888bdf94bb5db9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303977a3c371419fb23c244c6ee590c5" is_pre_election: true
I20260927 15:07:20.346076 7954 raft_consensus.cc:2454] T 43f54a8920f6409e88b3a4d0ec3812af P 303977a3c371419fb23c244c6ee590c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4888621aa8c04e24b888bdf94bb5db9d in term 0.
I20260927 15:07:20.346076 8222 raft_consensus.cc:2454] T 43f54a8920f6409e88b3a4d0ec3812af P 5cb8b28e8a444ffbbad0c387993b894a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4888621aa8c04e24b888bdf94bb5db9d in term 0.
I20260927 15:07:20.346575 8021 leader_election.cc:302] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4888621aa8c04e24b888bdf94bb5db9d, 5cb8b28e8a444ffbbad0c387993b894a; no voters:
I20260927 15:07:20.347292 8291 raft_consensus.cc:2781] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:20.347400 8291 raft_consensus.cc:482] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:20.347450 8291 raft_consensus.cc:3037] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:20.348488 8291 raft_consensus.cc:504] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303977a3c371419fb23c244c6ee590c5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41063 } } peers { permanent_uuid: "5cb8b28e8a444ffbbad0c387993b894a" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 45787 } } peers { permanent_uuid: "4888621aa8c04e24b888bdf94bb5db9d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39529 } }
I20260927 15:07:20.348903 8291 leader_election.cc:288] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d [CANDIDATE]: Term 1 election: Requested vote from peers 303977a3c371419fb23c244c6ee590c5 (127.7.17.193:41063), 5cb8b28e8a444ffbbad0c387993b894a (127.7.17.195:45787)
I20260927 15:07:20.349228 7954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f54a8920f6409e88b3a4d0ec3812af" candidate_uuid: "4888621aa8c04e24b888bdf94bb5db9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303977a3c371419fb23c244c6ee590c5"
I20260927 15:07:20.349241 8222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f54a8920f6409e88b3a4d0ec3812af" candidate_uuid: "4888621aa8c04e24b888bdf94bb5db9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb8b28e8a444ffbbad0c387993b894a"
I20260927 15:07:20.349371 8222 raft_consensus.cc:3037] T 43f54a8920f6409e88b3a4d0ec3812af P 5cb8b28e8a444ffbbad0c387993b894a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:20.349370 7954 raft_consensus.cc:3037] T 43f54a8920f6409e88b3a4d0ec3812af P 303977a3c371419fb23c244c6ee590c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:20.350406 7954 raft_consensus.cc:2454] T 43f54a8920f6409e88b3a4d0ec3812af P 303977a3c371419fb23c244c6ee590c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4888621aa8c04e24b888bdf94bb5db9d in term 1.
I20260927 15:07:20.350431 8222 raft_consensus.cc:2454] T 43f54a8920f6409e88b3a4d0ec3812af P 5cb8b28e8a444ffbbad0c387993b894a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4888621aa8c04e24b888bdf94bb5db9d in term 1.
I20260927 15:07:20.350706 8021 leader_election.cc:302] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4888621aa8c04e24b888bdf94bb5db9d, 5cb8b28e8a444ffbbad0c387993b894a; no voters:
I20260927 15:07:20.350935 8291 raft_consensus.cc:2781] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:20.351156 8291 raft_consensus.cc:686] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d [term 1 LEADER]: Becoming Leader. State: Replica: 4888621aa8c04e24b888bdf94bb5db9d, State: Running, Role: LEADER
I20260927 15:07:20.351467 8291 consensus_queue.cc:237] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303977a3c371419fb23c244c6ee590c5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41063 } } peers { permanent_uuid: "5cb8b28e8a444ffbbad0c387993b894a" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 45787 } } peers { permanent_uuid: "4888621aa8c04e24b888bdf94bb5db9d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39529 } }
I20260927 15:07:20.354725 7808 catalog_manager.cc:5495] T 43f54a8920f6409e88b3a4d0ec3812af P 4888621aa8c04e24b888bdf94bb5db9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4888621aa8c04e24b888bdf94bb5db9d (127.7.17.194). New cstate: current_term: 1 leader_uuid: "4888621aa8c04e24b888bdf94bb5db9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303977a3c371419fb23c244c6ee590c5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb8b28e8a444ffbbad0c387993b894a" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 45787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4888621aa8c04e24b888bdf94bb5db9d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39529 } health_report { overall_health: HEALTHY } } }
Sep 27 15:07:20 dist-test-slave-hhpq krb5kdc[7764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.193@KRBTEST.COM
I20260927 15:07:20.385409 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:20.376358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54496 (local address 127.7.17.193:41063)
0927 15:07:20.376520 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:20.376524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:20.376629 (+ 105us) server_negotiation.cc:407] Connection header received
0927 15:07:20.376738 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:20.376741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:20.376788 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:20.376917 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:20.377551 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.378085 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.378890 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.379116 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.379194 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:20.381597 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:20.381610 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:20.381626 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:20.381653 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:20.383346 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:20.383848 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:20.383854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:20.383878 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:20.384033 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:20.384355 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:20.384365 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:20.384367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:20.384668 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:20.384880 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:20.385099 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 15:07:20.385216 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 15:07:20 dist-test-slave-hhpq krb5kdc[7764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521640, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:07:20 dist-test-slave-hhpq krb5kdc[7764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521640, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.17.193@KRBTEST.COM
I20260927 15:07:20.413015 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:20.403220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54510 (local address 127.7.17.193:41063)
0927 15:07:20.403346 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:20.403351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:20.403576 (+ 225us) server_negotiation.cc:407] Connection header received
0927 15:07:20.403713 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:20.403716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:20.403763 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:20.403836 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:20.404547 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.405182 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.406036 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.406359 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.406472 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:20.409438 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:20.409442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:20.409445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:20.409493 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:20.411108 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:20.411618 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:20.411622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:20.411624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:20.411725 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:20.411987 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:20.411991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:20.411993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:20.412211 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:20.412336 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:20.412715 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 15:07:20.412866 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 15:07:20 dist-test-slave-hhpq krb5kdc[7764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:07:20 dist-test-slave-hhpq krb5kdc[7764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.193@KRBTEST.COM
I20260927 15:07:20.438825 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:20.430327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54522 (local address 127.7.17.193:41063)
0927 15:07:20.430466 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:20.430470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:20.430673 (+ 203us) server_negotiation.cc:407] Connection header received
0927 15:07:20.430785 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:20.430787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:20.430832 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:20.430917 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:20.431662 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.432160 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.433084 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:20.433296 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:20.433378 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:20.435900 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:20.435903 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:20.435905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:20.435932 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:20.437273 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:20.437689 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:20.437691 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:20.437693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:20.437767 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:20.438008 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:20.438010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:20.438011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:20.438201 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:20.438349 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:20.438604 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 15:07:20.438703 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 15:07:20.440320 7913 tablet_service.cc:2908] Rejecting scan request for tablet 43f54a8920f6409e88b3a4d0ec3812af: Uninitialized: clean time has not yet been initialized
I20260927 15:07:20.442523 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 7871
I20260927 15:07:20.447671 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 8005
W20260927 15:07:20.448323 8269 tablet.cc:2369] T 43f54a8920f6409e88b3a4d0ec3812af P 5cb8b28e8a444ffbbad0c387993b894a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:07:20.453053 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 8139
I20260927 15:07:20.457957 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 7777
2026-09-27T15:07:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2287 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:07:20 dist-test-slave-hhpq krb5kdc[8309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:07:20 dist-test-slave-hhpq krb5kdc[8309](info): set up 2 sockets
Sep 27 15:07:20 dist-test-slave-hhpq krb5kdc[8309](info): commencing operation
krb5kdc: starting...
W20260927 15:07:21.416455 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:07:21 dist-test-slave-hhpq krb5kdc[8309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:07:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:07:21.536798 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:43257
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:43257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:21.646108 8322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:21.646406 8322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:21.646467 8322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:21.650621 8322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:21.650698 8322 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 15:07:21.650723 8322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:21.650746 8322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:21.650768 8322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:21.650797 8322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:21.656328 8322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:43257
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:43257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:21.657519 8322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:21.658619 8322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:21.667572 8327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:21.667587 8330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:21.667649 8322 server_base.cc:1034] running on GCE node
W20260927 15:07:21.667587 8328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:21.668519 8322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:21.669843 8322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:21.671018 8322 hybrid_clock.cc:648] HybridClock initialized: now 1790521641671002 us; error 32 us; skew 500 ppm
Sep 27 15:07:21 dist-test-slave-hhpq krb5kdc[8309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521641, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:21.678682 8322 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:07:21.679744 8322 webserver.cc:460] Webserver started at http://127.7.17.254:44557/ using document root <none> and password file <none>
I20260927 15:07:21.680354 8322 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:21.680424 8322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:21.680694 8322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:21.683030 8322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "80af636a3cec4b16b03eb336950e2901"
format_stamp: "Formatted at 2026-09-27 15:07:21 on dist-test-slave-hhpq"
server_key: "8d22de121e6e9136a1725a92352b5cb4"
server_key_iv: "962ef6ed1ee9f9ad707f1749a1e38c55"
server_key_version: "encryptionkey@0"
I20260927 15:07:21.683595 8322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "80af636a3cec4b16b03eb336950e2901"
format_stamp: "Formatted at 2026-09-27 15:07:21 on dist-test-slave-hhpq"
server_key: "8d22de121e6e9136a1725a92352b5cb4"
server_key_iv: "962ef6ed1ee9f9ad707f1749a1e38c55"
server_key_version: "encryptionkey@0"
I20260927 15:07:21.687614 8322 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:07:21.690467 8337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:21.691902 8322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:07:21.692046 8322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "80af636a3cec4b16b03eb336950e2901"
format_stamp: "Formatted at 2026-09-27 15:07:21 on dist-test-slave-hhpq"
server_key: "8d22de121e6e9136a1725a92352b5cb4"
server_key_iv: "962ef6ed1ee9f9ad707f1749a1e38c55"
server_key_version: "encryptionkey@0"
I20260927 15:07:21.692163 8322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:21.703140 8322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:21.706393 8322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:21.706579 8322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:21.714248 8322 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:43257
I20260927 15:07:21.714267 8399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:43257 every 8 connection(s)
I20260927 15:07:21.715215 8322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:21.718180 8400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:21.723244 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 8322
I20260927 15:07:21.723438 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:07:21.723696 7239 external_mini_cluster.cc:1428] Setting key a708f4383444bb1c8b5870b81f01769e
I20260927 15:07:21.724481 8400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901: Bootstrap starting.
I20260927 15:07:21.726912 8400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:21.727843 8400 log.cc:824] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:21.730002 8400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901: No bootstrap required, opened a new log
Sep 27 15:07:21 dist-test-slave-hhpq krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:21.734261 8400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80af636a3cec4b16b03eb336950e2901" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43257 } }
I20260927 15:07:21.734519 8400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:21.734593 8400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80af636a3cec4b16b03eb336950e2901, State: Initialized, Role: FOLLOWER
I20260927 15:07:21.735054 8400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80af636a3cec4b16b03eb336950e2901" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43257 } }
I20260927 15:07:21.735175 8400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:21.735234 8400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:21.735332 8400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:21.736444 8400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80af636a3cec4b16b03eb336950e2901" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43257 } }
I20260927 15:07:21.736807 8400 leader_election.cc:302] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80af636a3cec4b16b03eb336950e2901; no voters:
I20260927 15:07:21.737188 8400 leader_election.cc:288] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:21.737308 8405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:21.737512 8405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901 [term 1 LEADER]: Becoming Leader. State: Replica: 80af636a3cec4b16b03eb336950e2901, State: Running, Role: LEADER
I20260927 15:07:21.737900 8405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80af636a3cec4b16b03eb336950e2901" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43257 } }
I20260927 15:07:21.738605 8400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:21.739316 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:21.725146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58236 (local address 127.7.17.254:43257)
0927 15:07:21.725576 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:21.725590 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:21.725625 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:07:21.726286 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:21.726304 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:21.726660 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:21.726965 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:21.727800 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:21.728892 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:21.730046 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:21.730606 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:21.730737 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:21.733578 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:21.733584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:21.733607 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:21.733641 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:21.735456 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:21.735956 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:21.735962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:21.735966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:21.736095 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:21.736441 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:21.736444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:21.736446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:21.737009 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:21.737205 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:21.737912 (+ 707us) server_negotiation.cc:299] Negotiation successful
0927 15:07:21.738232 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":120,"threads_started":1}
I20260927 15:07:21.739662 8407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80af636a3cec4b16b03eb336950e2901. Latest consensus state: current_term: 1 leader_uuid: "80af636a3cec4b16b03eb336950e2901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80af636a3cec4b16b03eb336950e2901" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43257 } } }
I20260927 15:07:21.739815 8407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:21.739686 8406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80af636a3cec4b16b03eb336950e2901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80af636a3cec4b16b03eb336950e2901" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43257 } } }
I20260927 15:07:21.740139 8406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:21.740202 8414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:21.743649 8414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:21.749445 8414 catalog_manager.cc:1347] Generated new cluster ID: ee2fde8d61954f08bc18896363b5b6ae
I20260927 15:07:21.749534 8414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:21.758858 8414 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:21.759855 8414 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:21.770919 8414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901: Generated new TSK 0
I20260927 15:07:21.771876 8414 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
I20260927 15:07:21.808269 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43257
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:21.922334 8426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:21.922636 8426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:21.922696 8426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:21.926851 8426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:21.926927 8426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:21.926982 8426 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:07:21.927034 8426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:07:21.932489 8426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.17.254:43257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:21.933704 8426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:21.934738 8426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:21.944875 8434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:21.944965 8426 server_base.cc:1034] running on GCE node
W20260927 15:07:21.945092 8432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:21.945307 8431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:21.945768 8426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:21.946466 8426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:21.947656 8426 hybrid_clock.cc:648] HybridClock initialized: now 1790521641947633 us; error 36 us; skew 500 ppm
Sep 27 15:07:21 dist-test-slave-hhpq krb5kdc[8309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521641, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:21.950590 8426 init.cc:377] Logged in from keytab as kudu/127.7.17.193@KRBTEST.COM (short username kudu)
I20260927 15:07:21.951519 8426 webserver.cc:460] Webserver started at http://127.7.17.193:45301/ using document root <none> and password file <none>
I20260927 15:07:21.952068 8426 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:21.952135 8426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:21.952342 8426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:21.954473 8426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "7352f4752a784da297ffff12dfc133d7"
format_stamp: "Formatted at 2026-09-27 15:07:21 on dist-test-slave-hhpq"
server_key: "79f0897fd9134036992ff7e64402fa8c"
server_key_iv: "0341b0bbf58079cd9e87c23c537d3d3c"
server_key_version: "encryptionkey@0"
I20260927 15:07:21.955034 8426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "7352f4752a784da297ffff12dfc133d7"
format_stamp: "Formatted at 2026-09-27 15:07:21 on dist-test-slave-hhpq"
server_key: "79f0897fd9134036992ff7e64402fa8c"
server_key_iv: "0341b0bbf58079cd9e87c23c537d3d3c"
server_key_version: "encryptionkey@0"
I20260927 15:07:21.958909 8426 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:07:21.961550 8441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:21.962906 8426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:21.963017 8426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "7352f4752a784da297ffff12dfc133d7"
format_stamp: "Formatted at 2026-09-27 15:07:21 on dist-test-slave-hhpq"
server_key: "79f0897fd9134036992ff7e64402fa8c"
server_key_iv: "0341b0bbf58079cd9e87c23c537d3d3c"
server_key_version: "encryptionkey@0"
I20260927 15:07:21.963145 8426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:21.987001 8426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:21.990521 8426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:21.990693 8426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:21.992123 8426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:21.992174 8426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:21.992228 8426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:21.992262 8426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:22.002195 8426 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:37311
I20260927 15:07:22.002241 8556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:37311 every 8 connection(s)
Sep 27 15:07:21 dist-test-slave-hhpq krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521641, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:22.003291 8426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:22.005182 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 8426
I20260927 15:07:22.005286 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:07:22.005575 7239 external_mini_cluster.cc:1428] Setting key 53daa355f3396a1cb305ddcc6e28d0a6
I20260927 15:07:22.006911 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:21.993030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:56303 (local address 127.7.17.254:43257)
0927 15:07:21.993146 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:21.993149 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:21.994236 (+ 1087us) server_negotiation.cc:407] Connection header received
0927 15:07:21.995416 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:21.995420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:21.995479 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:21.995538 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:21.997324 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:21.998126 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:21.999836 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.000099 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.000211 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.003200 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:22.003203 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:22.003206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:22.003232 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:22.004735 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:22.005243 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:22.005247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:22.005248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:22.005337 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:22.005748 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:22.005752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:22.005754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:22.005989 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:22.006089 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:22.006628 (+ 539us) server_negotiation.cc:299] Negotiation successful
0927 15:07:22.006755 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 15:07:22.007946 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:21.993618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:43257 (local address 127.7.17.193:56303)
0927 15:07:21.994088 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:21.994131 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:21.995104 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:21.995937 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:21.995944 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:21.996380 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:21.997137 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:21.997159 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:21.998261 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:21.998265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:21.999684 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:21.999696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.000257 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.000261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.000503 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.001199 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:22.001218 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:22.002996 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:22.004877 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:22.004883 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:22.004899 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:22.005143 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:22.005432 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:22.005435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:22.005437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:22.005624 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:22.006100 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:22.006108 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:22.006384 (+ 276us) client_negotiation.cc:769] Sending connection context
0927 15:07:22.006551 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 15:07:22.006898 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":133,"threads_started":1}
I20260927 15:07:22.009238 8557 heartbeater.cc:347] Connected to a master server at 127.7.17.254:43257
I20260927 15:07:22.009404 8557 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:22.009913 8557 heartbeater.cc:511] Master 127.7.17.254:43257 requested a full tablet report, sending...
I20260927 15:07:22.011439 8353 ts_manager.cc:194] Registered new tserver with Master: 7352f4752a784da297ffff12dfc133d7 (127.7.17.193:37311)
I20260927 15:07:22.012630 8353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:56303
WARNING: no policy specified for kudu/127.7.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.194@KRBTEST.COM" created.
I20260927 15:07:22.020324 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.011536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:43257 (local address 127.7.17.193:52415)
0927 15:07:22.013080 (+ 1544us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:22.013095 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:22.013205 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:22.013481 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:22.013484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:22.013713 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:22.013848 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.013856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.014844 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.014848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.015619 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.015628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.016082 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.016085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.016212 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.016753 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:22.016765 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:22.017119 (+ 354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:22.018864 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:22.018867 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:22.018870 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:22.019106 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:22.019442 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:22.019445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:22.019446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:22.019522 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:22.019894 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:22.019896 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:22.019978 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 15:07:22.020057 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 15:07:22.020172 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1457,"thread_start_us":103,"threads_started":1}
I20260927 15:07:22.020408 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.011605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:52415 (local address 127.7.17.254:43257)
0927 15:07:22.011719 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:22.011723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:22.013176 (+ 1453us) server_negotiation.cc:407] Connection header received
0927 15:07:22.013323 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:22.013327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:22.013374 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:22.013455 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:22.013974 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.014689 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.015732 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.015952 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.016036 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.017260 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:22.017264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:22.017266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:22.017298 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:22.018747 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:22.019236 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:22.019240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:22.019242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:22.019344 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:22.019590 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:22.019593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:22.019595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:22.019806 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:22.019902 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:22.020123 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 15:07:22.020258 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
I20260927 15:07:22.035404 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43257
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:22.146629 8566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:22.146936 8566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:22.147004 8566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:22.151207 8566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:22.151281 8566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:22.151340 8566 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:07:22.151403 8566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:07:22.157143 8566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.17.254:43257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:22.158509 8566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:22.159747 8566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:22.169625 8571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:22.169705 8572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:22.169687 8566 server_base.cc:1034] running on GCE node
W20260927 15:07:22.169780 8574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:22.170270 8566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:22.170925 8566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:22.172071 8566 hybrid_clock.cc:648] HybridClock initialized: now 1790521642172044 us; error 43 us; skew 500 ppm
Sep 27 15:07:22 dist-test-slave-hhpq krb5kdc[8309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521642, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:22.174736 8566 init.cc:377] Logged in from keytab as kudu/127.7.17.194@KRBTEST.COM (short username kudu)
I20260927 15:07:22.175639 8566 webserver.cc:460] Webserver started at http://127.7.17.194:45229/ using document root <none> and password file <none>
I20260927 15:07:22.176245 8566 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:22.176313 8566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:22.176499 8566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:22.178601 8566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "910591e01dfa488fabd5a14d0a764c3a"
format_stamp: "Formatted at 2026-09-27 15:07:22 on dist-test-slave-hhpq"
server_key: "77d0374844154d2054c445e13c97467a"
server_key_iv: "ece6475de0674127b52bf4627a3399d5"
server_key_version: "encryptionkey@0"
I20260927 15:07:22.179159 8566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "910591e01dfa488fabd5a14d0a764c3a"
format_stamp: "Formatted at 2026-09-27 15:07:22 on dist-test-slave-hhpq"
server_key: "77d0374844154d2054c445e13c97467a"
server_key_iv: "ece6475de0674127b52bf4627a3399d5"
server_key_version: "encryptionkey@0"
I20260927 15:07:22.183076 8566 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:07:22.185652 8581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:22.186954 8566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:07:22.187064 8566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "910591e01dfa488fabd5a14d0a764c3a"
format_stamp: "Formatted at 2026-09-27 15:07:22 on dist-test-slave-hhpq"
server_key: "77d0374844154d2054c445e13c97467a"
server_key_iv: "ece6475de0674127b52bf4627a3399d5"
server_key_version: "encryptionkey@0"
I20260927 15:07:22.187165 8566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:22.206831 8566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:22.209978 8566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:22.210150 8566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:22.211549 8566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:22.211598 8566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:22.211649 8566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:22.211676 8566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:07:22 dist-test-slave-hhpq krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521642, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:22.221928 8566 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:39493
I20260927 15:07:22.221974 8696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:39493 every 8 connection(s)
I20260927 15:07:22.223043 8566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:07:22.225443 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.212343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:36661 (local address 127.7.17.254:43257)
0927 15:07:22.212475 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:22.212478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:22.213609 (+ 1131us) server_negotiation.cc:407] Connection header received
0927 15:07:22.214628 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:22.214632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:22.214688 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:22.214733 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:22.216093 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.216639 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.217816 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.218040 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.218132 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.221314 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:22.221317 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:22.221327 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:22.221354 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:22.223076 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:22.223719 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:22.223723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:22.223726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:22.223829 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:22.224307 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:22.224311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:22.224312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:22.224513 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:22.224596 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:22.225144 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 15:07:22.225268 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:07:22.226218 8590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.212967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:43257 (local address 127.7.17.194:36661)
0927 15:07:22.213449 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:22.213494 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:22.214381 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:22.215053 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:22.215059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:22.215384 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:22.215952 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.215969 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.216762 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.216765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.217694 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.217704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.218235 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.218240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.218468 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.219129 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:22.219150 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:22.221121 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:22.223219 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:22.223225 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:22.223240 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:22.223596 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:22.224082 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:22.224085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:22.224086 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:22.224209 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:22.224614 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:22.224621 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:22.224916 (+ 295us) client_negotiation.cc:769] Sending connection context
0927 15:07:22.225074 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 15:07:22.225398 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":123,"threads_started":1}
I20260927 15:07:22.227550 8697 heartbeater.cc:347] Connected to a master server at 127.7.17.254:43257
I20260927 15:07:22.227703 8697 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:22.228289 8697 heartbeater.cc:511] Master 127.7.17.254:43257 requested a full tablet report, sending...
I20260927 15:07:22.229483 8354 ts_manager.cc:194] Registered new tserver with Master: 910591e01dfa488fabd5a14d0a764c3a (127.7.17.194:39493)
I20260927 15:07:22.230162 8354 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:36661
I20260927 15:07:22.231806 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 8566
I20260927 15:07:22.231931 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:07:22.232218 7239 external_mini_cluster.cc:1428] Setting key 5dfa1d626e3f670a7eee6fcb16bd6c50
I20260927 15:07:22.237279 8703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.229858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:43257 (local address 127.7.17.194:39217)
0927 15:07:22.230196 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:22.230208 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:22.230306 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:22.230549 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:22.230551 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:22.230727 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:22.230839 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.230845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.231581 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.231584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.232233 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.232242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.232907 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.232910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.232982 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.233455 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:22.233467 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:22.233913 (+ 446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:22.235593 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:22.235596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:22.235599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:22.235839 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:22.236148 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:22.236150 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:22.236151 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:22.236230 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:22.236747 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:22.236749 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:22.236881 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 15:07:22.236988 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 15:07:22.237119 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":99,"threads_started":1}
I20260927 15:07:22.237325 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.229937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:39217 (local address 127.7.17.254:43257)
0927 15:07:22.230063 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:22.230068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:22.230282 (+ 214us) server_negotiation.cc:407] Connection header received
0927 15:07:22.230389 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:22.230392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:22.230435 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:22.230508 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:22.230953 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.231458 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.232377 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.232662 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.232751 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.234095 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:22.234098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:22.234100 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:22.234123 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:22.235488 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:22.235971 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:22.235975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:22.235976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:22.236067 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:22.236343 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:22.236347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:22.236349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:22.236626 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:22.236727 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:22.237052 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 15:07:22.237174 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.7.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
I20260927 15:07:22.260149 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43257
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:22.371095 8706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:22.371385 8706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:22.371450 8706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:22.375517 8706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:22.375586 8706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:22.375643 8706 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:07:22.375700 8706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:07:22.381206 8706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.17.254:43257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:22.382442 8706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:22.383569 8706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:22.393282 8712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:22.393380 8706 server_base.cc:1034] running on GCE node
W20260927 15:07:22.393304 8711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:22.393275 8714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:22.393916 8706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:22.394582 8706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:22.395737 8706 hybrid_clock.cc:648] HybridClock initialized: now 1790521642395717 us; error 33 us; skew 500 ppm
Sep 27 15:07:22 dist-test-slave-hhpq krb5kdc[8309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521642, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:22.398625 8706 init.cc:377] Logged in from keytab as kudu/127.7.17.195@KRBTEST.COM (short username kudu)
I20260927 15:07:22.399551 8706 webserver.cc:460] Webserver started at http://127.7.17.195:35283/ using document root <none> and password file <none>
I20260927 15:07:22.400130 8706 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:22.400195 8706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:22.400388 8706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:22.402536 8706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "fc0870906b0f47b3acc65d63e1f9e04c"
format_stamp: "Formatted at 2026-09-27 15:07:22 on dist-test-slave-hhpq"
server_key: "3a223f09cb10a80fe8ad2f959eefb05d"
server_key_iv: "155ab2435ae02912e7ba9187ccdeeeeb"
server_key_version: "encryptionkey@0"
I20260927 15:07:22.403079 8706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "fc0870906b0f47b3acc65d63e1f9e04c"
format_stamp: "Formatted at 2026-09-27 15:07:22 on dist-test-slave-hhpq"
server_key: "3a223f09cb10a80fe8ad2f959eefb05d"
server_key_iv: "155ab2435ae02912e7ba9187ccdeeeeb"
server_key_version: "encryptionkey@0"
I20260927 15:07:22.407250 8706 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:07:22.410462 8721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:22.411954 8706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:07:22.412065 8706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "fc0870906b0f47b3acc65d63e1f9e04c"
format_stamp: "Formatted at 2026-09-27 15:07:22 on dist-test-slave-hhpq"
server_key: "3a223f09cb10a80fe8ad2f959eefb05d"
server_key_iv: "155ab2435ae02912e7ba9187ccdeeeeb"
server_key_version: "encryptionkey@0"
I20260927 15:07:22.412186 8706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:22.452140 8706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:22.455433 8706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:22.455608 8706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:22.457173 8706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:22.457226 8706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:22.457265 8706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:22.457289 8706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:07:22 dist-test-slave-hhpq krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521642, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:22.467654 8706 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:35085
I20260927 15:07:22.467729 8836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:35085 every 8 connection(s)
I20260927 15:07:22.468636 8706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:22.471365 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.458019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:38995 (local address 127.7.17.254:43257)
0927 15:07:22.458135 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:22.458139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:22.459265 (+ 1126us) server_negotiation.cc:407] Connection header received
0927 15:07:22.460432 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:22.460436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:22.460498 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:22.460557 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:22.462190 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.462803 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.464599 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.464862 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.464958 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.467981 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:22.467986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:22.467989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:22.468019 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:22.469584 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:22.470062 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:22.470065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:22.470067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:22.470143 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:22.470400 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:22.470403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:22.470404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:22.470589 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:22.470665 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:22.471141 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 15:07:22.471244 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:07:22.472254 8730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.458540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:43257 (local address 127.7.17.195:38995)
0927 15:07:22.459090 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:22.459152 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:22.460160 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:22.461039 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:22.461046 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:22.461502 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:22.462038 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.462052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.462941 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.462945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.464463 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.464475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.465020 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.465025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.465244 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.465997 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:22.466017 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:22.467737 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:22.469694 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:22.469698 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:22.469709 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:22.469966 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:22.470209 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:22.470211 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:22.470212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:22.470335 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:22.470674 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:22.470681 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:22.470934 (+ 253us) client_negotiation.cc:769] Sending connection context
0927 15:07:22.471090 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 15:07:22.471358 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":126,"threads_started":1}
I20260927 15:07:22.473645 8837 heartbeater.cc:347] Connected to a master server at 127.7.17.254:43257
I20260927 15:07:22.473788 8837 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:22.474387 8837 heartbeater.cc:511] Master 127.7.17.254:43257 requested a full tablet report, sending...
I20260927 15:07:22.475446 8354 ts_manager.cc:194] Registered new tserver with Master: fc0870906b0f47b3acc65d63e1f9e04c (127.7.17.195:35085)
I20260927 15:07:22.476181 8354 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:38995
I20260927 15:07:22.477133 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 8706
I20260927 15:07:22.477245 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:07:22.477520 7239 external_mini_cluster.cc:1428] Setting key 10081523e13a8225c28705bfb4c59a77
I20260927 15:07:22.479653 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:07:22.483070 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.475711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:54303 (local address 127.7.17.254:43257)
0927 15:07:22.475830 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:22.475835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:22.476080 (+ 245us) server_negotiation.cc:407] Connection header received
0927 15:07:22.476164 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:22.476167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:22.476210 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:22.476290 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:22.476848 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.477427 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.478410 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.478666 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.478746 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.480034 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:22.480038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:22.480041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:22.480067 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:22.481524 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:22.481953 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:22.481956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:22.481958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:22.482037 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:22.482289 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:22.482293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:22.482294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:22.482476 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:22.482559 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:22.482804 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 15:07:22.482920 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:07:22.483083 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.475641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:43257 (local address 127.7.17.195:54303)
0927 15:07:22.475979 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:22.475992 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:22.476085 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:22.476369 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:22.476372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:22.476587 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:22.476708 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.476716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.477540 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.477554 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.478272 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.478292 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.478896 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.478900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.478993 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.479538 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:22.479552 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:22.479916 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:22.481642 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:22.481646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:22.481648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:22.481858 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:22.482123 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:22.482126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:22.482129 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:22.482209 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:22.482560 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:22.482564 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:22.482657 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 15:07:22.482741 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 15:07:22.482895 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":88,"threads_started":1}
Sep 27 15:07:22 dist-test-slave-hhpq krb5kdc[8309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521642, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:07:22 dist-test-slave-hhpq krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521642, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:22.505009 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.496568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.7.17.254:43257)
0927 15:07:22.496705 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:22.496709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:22.496724 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:07:22.496769 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:22.496771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:22.496818 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:22.496913 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:22.497681 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.498235 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.499186 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.499420 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.499506 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.501994 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:22.501998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:22.502000 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:22.502029 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:22.503406 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:22.503827 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:22.503830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:22.503831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:22.503903 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:22.504185 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:22.504187 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:22.504188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:22.504402 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:22.504502 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:22.504699 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 15:07:22.504795 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:07:22.507992 8354 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:07:22.511349 8354 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:22.526626 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.521035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34576 (local address 127.7.17.193:37311)
0927 15:07:22.521560 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:22.521567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:22.521590 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:07:22.521667 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:22.521673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:22.521895 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:22.522570 (+ 675us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:22.522696 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.523878 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.525334 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.525951 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.526053 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.526215 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:22.526366 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 15:07:22.526471 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":79,"threads_started":1}
I20260927 15:07:22.526628 8855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.520226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:37311 (local address 127.0.0.1:34576)
0927 15:07:22.520734 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:22.520787 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:22.520961 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:22.522069 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:22.522076 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:22.522112 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:22.522288 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.522296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.524007 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.524010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.525223 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.525230 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.526065 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.526069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.526174 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.526210 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:07:22.526333 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 15:07:22.526471 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":50,"spinlock_wait_cycles":5120,"thread_start_us":128,"threads_started":1}
I20260927 15:07:22.526641 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.521104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:35085 (local address 127.0.0.1:56444)
0927 15:07:22.521332 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:22.521345 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:22.521435 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:22.522069 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:22.522076 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:22.522110 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:22.522531 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.522542 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.523596 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.523599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.525154 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.525164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.526237 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.526241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.526338 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.526366 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 15:07:22.526412 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 15:07:22.526485 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":443,"spinlock_wait_cycles":5632,"thread_start_us":95,"threads_started":1}
I20260927 15:07:22.527097 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.520327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:39493 (local address 127.0.0.1:51590)
0927 15:07:22.521179 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:22.521192 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:22.521266 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:22.522417 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:22.522419 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:22.522436 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:22.522567 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.522573 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.524031 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.524035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.525297 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.525356 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.526393 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.526396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.526480 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.526498 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 15:07:22.526549 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 15:07:22.526603 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":108,"thread_start_us":120,"threads_started":1}
I20260927 15:07:22.527210 8856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.520564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51590 (local address 127.7.17.194:39493)
0927 15:07:22.521955 (+ 1391us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:22.521966 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:22.521991 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:07:22.522066 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:22.522071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:22.522269 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:22.522420 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:22.522694 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.523879 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.525476 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.526259 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.526736 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.526932 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:22.526987 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 15:07:22.527084 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1284,"thread_start_us":100,"threads_started":1}
I20260927 15:07:22.527727 8859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.521397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.7.17.195:35085)
0927 15:07:22.521669 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:22.521679 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:22.521697 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:07:22.521753 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:22.521767 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:22.521902 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:22.522187 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:22.522649 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.523466 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.525287 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.526097 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.527333 (+ 1236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.527497 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:22.527539 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 15:07:22.527611 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":99,"threads_started":1}
I20260927 15:07:22.530318 8771 tablet_service.cc:1467] Processing CreateTablet for tablet 2caf66395f9245a9bd86e07fd94a66ab (DEFAULT_TABLE table=test-table [id=5c8e3c72d6f14bf9a47fe7b4133214ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:22.530328 8631 tablet_service.cc:1467] Processing CreateTablet for tablet 2caf66395f9245a9bd86e07fd94a66ab (DEFAULT_TABLE table=test-table [id=5c8e3c72d6f14bf9a47fe7b4133214ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:22.530320 8491 tablet_service.cc:1467] Processing CreateTablet for tablet 2caf66395f9245a9bd86e07fd94a66ab (DEFAULT_TABLE table=test-table [id=5c8e3c72d6f14bf9a47fe7b4133214ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:22.531407 8771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2caf66395f9245a9bd86e07fd94a66ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:22.531402 8491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2caf66395f9245a9bd86e07fd94a66ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:22.531402 8631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2caf66395f9245a9bd86e07fd94a66ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:22.538549 8860 tablet_bootstrap.cc:492] T 2caf66395f9245a9bd86e07fd94a66ab P fc0870906b0f47b3acc65d63e1f9e04c: Bootstrap starting.
I20260927 15:07:22.539834 8861 tablet_bootstrap.cc:492] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7: Bootstrap starting.
I20260927 15:07:22.541015 8860 tablet_bootstrap.cc:654] T 2caf66395f9245a9bd86e07fd94a66ab P fc0870906b0f47b3acc65d63e1f9e04c: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:22.541735 8861 tablet_bootstrap.cc:654] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:22.541776 8860 log.cc:824] T 2caf66395f9245a9bd86e07fd94a66ab P fc0870906b0f47b3acc65d63e1f9e04c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:22.542477 8861 log.cc:824] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:22.543095 8862 tablet_bootstrap.cc:492] T 2caf66395f9245a9bd86e07fd94a66ab P 910591e01dfa488fabd5a14d0a764c3a: Bootstrap starting.
I20260927 15:07:22.544711 8861 tablet_bootstrap.cc:492] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7: No bootstrap required, opened a new log
I20260927 15:07:22.544952 8861 ts_tablet_manager.cc:1397] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 15:07:22.545090 8862 tablet_bootstrap.cc:654] T 2caf66395f9245a9bd86e07fd94a66ab P 910591e01dfa488fabd5a14d0a764c3a: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:22.545921 8862 log.cc:824] T 2caf66395f9245a9bd86e07fd94a66ab P 910591e01dfa488fabd5a14d0a764c3a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:22.546260 8860 tablet_bootstrap.cc:492] T 2caf66395f9245a9bd86e07fd94a66ab P fc0870906b0f47b3acc65d63e1f9e04c: No bootstrap required, opened a new log
I20260927 15:07:22.546499 8860 ts_tablet_manager.cc:1397] T 2caf66395f9245a9bd86e07fd94a66ab P fc0870906b0f47b3acc65d63e1f9e04c: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260927 15:07:22.547973 8862 tablet_bootstrap.cc:492] T 2caf66395f9245a9bd86e07fd94a66ab P 910591e01dfa488fabd5a14d0a764c3a: No bootstrap required, opened a new log
I20260927 15:07:22.548200 8862 ts_tablet_manager.cc:1397] T 2caf66395f9245a9bd86e07fd94a66ab P 910591e01dfa488fabd5a14d0a764c3a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260927 15:07:22.548784 8861 raft_consensus.cc:348] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.549098 8861 raft_consensus.cc:374] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:22.549172 8861 raft_consensus.cc:729] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7352f4752a784da297ffff12dfc133d7, State: Initialized, Role: FOLLOWER
I20260927 15:07:22.549629 8861 consensus_queue.cc:260] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.550285 8557 heartbeater.cc:503] Master 127.7.17.254:43257 was elected leader, sending a full tablet report...
I20260927 15:07:22.550577 8861 ts_tablet_manager.cc:1428] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:07:22.551889 8860 raft_consensus.cc:348] T 2caf66395f9245a9bd86e07fd94a66ab P fc0870906b0f47b3acc65d63e1f9e04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.551918 8862 raft_consensus.cc:348] T 2caf66395f9245a9bd86e07fd94a66ab P 910591e01dfa488fabd5a14d0a764c3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.552174 8860 raft_consensus.cc:374] T 2caf66395f9245a9bd86e07fd94a66ab P fc0870906b0f47b3acc65d63e1f9e04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:22.552175 8862 raft_consensus.cc:374] T 2caf66395f9245a9bd86e07fd94a66ab P 910591e01dfa488fabd5a14d0a764c3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:22.552232 8862 raft_consensus.cc:729] T 2caf66395f9245a9bd86e07fd94a66ab P 910591e01dfa488fabd5a14d0a764c3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910591e01dfa488fabd5a14d0a764c3a, State: Initialized, Role: FOLLOWER
I20260927 15:07:22.552317 8860 raft_consensus.cc:729] T 2caf66395f9245a9bd86e07fd94a66ab P fc0870906b0f47b3acc65d63e1f9e04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc0870906b0f47b3acc65d63e1f9e04c, State: Initialized, Role: FOLLOWER
I20260927 15:07:22.552738 8860 consensus_queue.cc:260] T 2caf66395f9245a9bd86e07fd94a66ab P fc0870906b0f47b3acc65d63e1f9e04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.552738 8862 consensus_queue.cc:260] T 2caf66395f9245a9bd86e07fd94a66ab P 910591e01dfa488fabd5a14d0a764c3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.553512 8697 heartbeater.cc:503] Master 127.7.17.254:43257 was elected leader, sending a full tablet report...
I20260927 15:07:22.553766 8837 heartbeater.cc:503] Master 127.7.17.254:43257 was elected leader, sending a full tablet report...
I20260927 15:07:22.554946 8860 ts_tablet_manager.cc:1428] T 2caf66395f9245a9bd86e07fd94a66ab P fc0870906b0f47b3acc65d63e1f9e04c: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260927 15:07:22.555209 8862 ts_tablet_manager.cc:1428] T 2caf66395f9245a9bd86e07fd94a66ab P 910591e01dfa488fabd5a14d0a764c3a: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260927 15:07:22.719674 8838 tablet.cc:2369] T 2caf66395f9245a9bd86e07fd94a66ab P fc0870906b0f47b3acc65d63e1f9e04c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:07:22.724226 8698 tablet.cc:2369] T 2caf66395f9245a9bd86e07fd94a66ab P 910591e01dfa488fabd5a14d0a764c3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:07:22.754662 8558 tablet.cc:2369] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:07:22.760473 8866 raft_consensus.cc:482] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:22.760690 8866 raft_consensus.cc:504] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.761920 8866 leader_election.cc:288] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 910591e01dfa488fabd5a14d0a764c3a (127.7.17.194:39493), fc0870906b0f47b3acc65d63e1f9e04c (127.7.17.195:35085)
I20260927 15:07:22.765532 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.762138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:39493 (local address 127.7.17.193:35085)
0927 15:07:22.762414 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:22.762427 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:22.762539 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:22.762791 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:22.762795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:22.762814 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:22.762935 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.762941 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.763646 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.763649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.764450 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.764458 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.765124 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.765127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.765221 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.765239 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 15:07:22.765307 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 15:07:22.765378 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":98,"threads_started":1}
I20260927 15:07:22.765518 8856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.762219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:35085 (local address 127.7.17.194:39493)
0927 15:07:22.762356 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:22.762360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:22.762514 (+ 154us) server_negotiation.cc:407] Connection header received
0927 15:07:22.762645 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:22.762648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:22.762703 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:22.762781 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:22.763030 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.763542 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.764562 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.765018 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.765118 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.765170 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:22.765332 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 15:07:22.765402 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:07:22.765851 8859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.762654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:52253 (local address 127.7.17.195:35085)
0927 15:07:22.762771 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:22.762775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:22.762967 (+ 192us) server_negotiation.cc:407] Connection header received
0927 15:07:22.763066 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:22.763069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:22.763110 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:22.763169 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:22.763410 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.763908 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.764928 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.765309 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.765422 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.765469 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:22.765654 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 15:07:22.765731 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:07:22.766201 8870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.762532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:35085 (local address 127.7.17.193:52253)
0927 15:07:22.762889 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:22.762899 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:22.762987 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:22.763192 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:22.763194 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:22.763208 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:22.763315 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.763321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.764023 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.764027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.764775 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.764783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.765423 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.765426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.765519 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.765541 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:07:22.766034 (+ 493us) client_negotiation.cc:240] Negotiation successful
0927 15:07:22.766094 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":92,"threads_started":1}
I20260927 15:07:22.766153 8651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2caf66395f9245a9bd86e07fd94a66ab" candidate_uuid: "7352f4752a784da297ffff12dfc133d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "910591e01dfa488fabd5a14d0a764c3a" is_pre_election: true
I20260927 15:07:22.766484 8651 raft_consensus.cc:2454] T 2caf66395f9245a9bd86e07fd94a66ab P 910591e01dfa488fabd5a14d0a764c3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7352f4752a784da297ffff12dfc133d7 in term 0.
I20260927 15:07:22.766888 8791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2caf66395f9245a9bd86e07fd94a66ab" candidate_uuid: "7352f4752a784da297ffff12dfc133d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" is_pre_election: true
I20260927 15:07:22.767023 8442 leader_election.cc:302] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7352f4752a784da297ffff12dfc133d7, 910591e01dfa488fabd5a14d0a764c3a; no voters:
I20260927 15:07:22.767197 8791 raft_consensus.cc:2454] T 2caf66395f9245a9bd86e07fd94a66ab P fc0870906b0f47b3acc65d63e1f9e04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7352f4752a784da297ffff12dfc133d7 in term 0.
I20260927 15:07:22.767344 8866 raft_consensus.cc:2781] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:22.767428 8866 raft_consensus.cc:482] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:22.767480 8866 raft_consensus.cc:3037] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:22.768574 8866 raft_consensus.cc:504] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.768932 8866 leader_election.cc:288] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7 [CANDIDATE]: Term 1 election: Requested vote from peers 910591e01dfa488fabd5a14d0a764c3a (127.7.17.194:39493), fc0870906b0f47b3acc65d63e1f9e04c (127.7.17.195:35085)
I20260927 15:07:22.769307 8651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2caf66395f9245a9bd86e07fd94a66ab" candidate_uuid: "7352f4752a784da297ffff12dfc133d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "910591e01dfa488fabd5a14d0a764c3a"
I20260927 15:07:22.769381 8791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2caf66395f9245a9bd86e07fd94a66ab" candidate_uuid: "7352f4752a784da297ffff12dfc133d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc0870906b0f47b3acc65d63e1f9e04c"
I20260927 15:07:22.769429 8651 raft_consensus.cc:3037] T 2caf66395f9245a9bd86e07fd94a66ab P 910591e01dfa488fabd5a14d0a764c3a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:22.769510 8791 raft_consensus.cc:3037] T 2caf66395f9245a9bd86e07fd94a66ab P fc0870906b0f47b3acc65d63e1f9e04c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:22.770471 8651 raft_consensus.cc:2454] T 2caf66395f9245a9bd86e07fd94a66ab P 910591e01dfa488fabd5a14d0a764c3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7352f4752a784da297ffff12dfc133d7 in term 1.
I20260927 15:07:22.770797 8442 leader_election.cc:302] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7352f4752a784da297ffff12dfc133d7, 910591e01dfa488fabd5a14d0a764c3a; no voters:
I20260927 15:07:22.771010 8791 raft_consensus.cc:2454] T 2caf66395f9245a9bd86e07fd94a66ab P fc0870906b0f47b3acc65d63e1f9e04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7352f4752a784da297ffff12dfc133d7 in term 1.
I20260927 15:07:22.771027 8866 raft_consensus.cc:2781] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:22.771270 8866 raft_consensus.cc:686] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7 [term 1 LEADER]: Becoming Leader. State: Replica: 7352f4752a784da297ffff12dfc133d7, State: Running, Role: LEADER
I20260927 15:07:22.771656 8866 consensus_queue.cc:237] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.775075 8353 catalog_manager.cc:5495] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7352f4752a784da297ffff12dfc133d7 (127.7.17.193). New cstate: current_term: 1 leader_uuid: "7352f4752a784da297ffff12dfc133d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } health_report { overall_health: UNKNOWN } } }
I20260927 15:07:22.832384 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.828740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34588 (local address 127.7.17.193:37311)
0927 15:07:22.828901 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:22.828906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:22.828923 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:07:22.829106 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:22.829108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:22.829162 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:22.829246 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:22.829509 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.830022 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.830901 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.831106 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.831180 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.831562 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:22.832008 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:22.832088 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:22.832196 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 15:07:22.832247 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:07:22.838495 8791 raft_consensus.cc:1261] T 2caf66395f9245a9bd86e07fd94a66ab P fc0870906b0f47b3acc65d63e1f9e04c [term 1 FOLLOWER]: Refusing update from remote peer 7352f4752a784da297ffff12dfc133d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:22.838515 8651 raft_consensus.cc:1261] T 2caf66395f9245a9bd86e07fd94a66ab P 910591e01dfa488fabd5a14d0a764c3a [term 1 FOLLOWER]: Refusing update from remote peer 7352f4752a784da297ffff12dfc133d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:22.839184 8866 consensus_queue.cc:1035] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:22.839427 8871 consensus_queue.cc:1035] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:22.846175 8874 mvcc.cc:205] Tried to move back new op lower bound from 7333976649058398208 to 7333976648795385856. Current Snapshot: MvccSnapshot[applied={T|T < 7333976649058398208}]
I20260927 15:07:22.848438 8875 mvcc.cc:205] Tried to move back new op lower bound from 7333976649058398208 to 7333976648795385856. Current Snapshot: MvccSnapshot[applied={T|T < 7333976649058398208}]
I20260927 15:07:22.858332 8354 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:22.858502 8354 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:22.860934 8354 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901: Sending DeleteTablet for 3 replicas of tablet 2caf66395f9245a9bd86e07fd94a66ab
I20260927 15:07:22.861734 8631 tablet_service.cc:1514] Processing DeleteTablet for tablet 2caf66395f9245a9bd86e07fd94a66ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:22 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:51590
I20260927 15:07:22.861716 8491 tablet_service.cc:1514] Processing DeleteTablet for tablet 2caf66395f9245a9bd86e07fd94a66ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:22 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:34576
I20260927 15:07:22.861810 8771 tablet_service.cc:1514] Processing DeleteTablet for tablet 2caf66395f9245a9bd86e07fd94a66ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:22 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:56444
I20260927 15:07:22.862365 8883 tablet_replica.cc:330] T 2caf66395f9245a9bd86e07fd94a66ab P 910591e01dfa488fabd5a14d0a764c3a: stopping tablet replica
I20260927 15:07:22.862437 8885 tablet_replica.cc:330] T 2caf66395f9245a9bd86e07fd94a66ab P fc0870906b0f47b3acc65d63e1f9e04c: stopping tablet replica
I20260927 15:07:22.862799 8885 raft_consensus.cc:2229] T 2caf66395f9245a9bd86e07fd94a66ab P fc0870906b0f47b3acc65d63e1f9e04c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:22.862798 8883 raft_consensus.cc:2229] T 2caf66395f9245a9bd86e07fd94a66ab P 910591e01dfa488fabd5a14d0a764c3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:22.863176 8885 raft_consensus.cc:2258] T 2caf66395f9245a9bd86e07fd94a66ab P fc0870906b0f47b3acc65d63e1f9e04c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:22.863199 8883 raft_consensus.cc:2258] T 2caf66395f9245a9bd86e07fd94a66ab P 910591e01dfa488fabd5a14d0a764c3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:22.863279 8354 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 15:07:22.863559 8884 tablet_replica.cc:330] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7: stopping tablet replica
W20260927 15:07:22.863730 8354 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:22.863826 8884 raft_consensus.cc:2229] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:07:22.864223 8884 raft_consensus.cc:2258] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:22.865666 8884 ts_tablet_manager.cc:1905] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:22.866016 8885 ts_tablet_manager.cc:1905] T 2caf66395f9245a9bd86e07fd94a66ab P fc0870906b0f47b3acc65d63e1f9e04c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:22.866154 8883 ts_tablet_manager.cc:1905] T 2caf66395f9245a9bd86e07fd94a66ab P 910591e01dfa488fabd5a14d0a764c3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:22.868999 8885 ts_tablet_manager.cc:1918] T 2caf66395f9245a9bd86e07fd94a66ab P fc0870906b0f47b3acc65d63e1f9e04c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:22.869171 8885 log.cc:1196] T 2caf66395f9245a9bd86e07fd94a66ab P fc0870906b0f47b3acc65d63e1f9e04c: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/2caf66395f9245a9bd86e07fd94a66ab
I20260927 15:07:22.869587 8885 ts_tablet_manager.cc:1939] T 2caf66395f9245a9bd86e07fd94a66ab P fc0870906b0f47b3acc65d63e1f9e04c: Deleting consensus metadata
I20260927 15:07:22.870095 8631 tablet_service.cc:1467] Processing CreateTablet for tablet b270ae93a0064fdeb230978e02f8b07d (DEFAULT_TABLE table=test-table [id=98b7a1beba9546338faa27a4229f5c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:22.870357 8631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b270ae93a0064fdeb230978e02f8b07d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:22.871161 8771 tablet_service.cc:1467] Processing CreateTablet for tablet b270ae93a0064fdeb230978e02f8b07d (DEFAULT_TABLE table=test-table [id=98b7a1beba9546338faa27a4229f5c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:22.871399 8771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b270ae93a0064fdeb230978e02f8b07d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:22.871744 8884 ts_tablet_manager.cc:1918] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:22.871832 8884 log.cc:1196] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-0/wal/wals/2caf66395f9245a9bd86e07fd94a66ab
I20260927 15:07:22.872201 8884 ts_tablet_manager.cc:1939] T 2caf66395f9245a9bd86e07fd94a66ab P 7352f4752a784da297ffff12dfc133d7: Deleting consensus metadata
I20260927 15:07:22.872512 8862 tablet_bootstrap.cc:492] T b270ae93a0064fdeb230978e02f8b07d P 910591e01dfa488fabd5a14d0a764c3a: Bootstrap starting.
I20260927 15:07:22.872779 8338 catalog_manager.cc:4841] TS fc0870906b0f47b3acc65d63e1f9e04c (127.7.17.195:35085): tablet 2caf66395f9245a9bd86e07fd94a66ab (table test-table [id=5c8e3c72d6f14bf9a47fe7b4133214ad]) successfully deleted
I20260927 15:07:22.870136 8883 ts_tablet_manager.cc:1918] T 2caf66395f9245a9bd86e07fd94a66ab P 910591e01dfa488fabd5a14d0a764c3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:22.873420 8860 tablet_bootstrap.cc:492] T b270ae93a0064fdeb230978e02f8b07d P fc0870906b0f47b3acc65d63e1f9e04c: Bootstrap starting.
I20260927 15:07:22.873441 8862 tablet_bootstrap.cc:654] T b270ae93a0064fdeb230978e02f8b07d P 910591e01dfa488fabd5a14d0a764c3a: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:22.873977 8340 catalog_manager.cc:4841] TS 7352f4752a784da297ffff12dfc133d7 (127.7.17.193:37311): tablet 2caf66395f9245a9bd86e07fd94a66ab (table test-table [id=5c8e3c72d6f14bf9a47fe7b4133214ad]) successfully deleted
I20260927 15:07:22.874284 8860 tablet_bootstrap.cc:654] T b270ae93a0064fdeb230978e02f8b07d P fc0870906b0f47b3acc65d63e1f9e04c: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:22.874529 8491 tablet_service.cc:1467] Processing CreateTablet for tablet b270ae93a0064fdeb230978e02f8b07d (DEFAULT_TABLE table=test-table [id=98b7a1beba9546338faa27a4229f5c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:22.874768 8491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b270ae93a0064fdeb230978e02f8b07d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:22.874827 8883 log.cc:1196] T 2caf66395f9245a9bd86e07fd94a66ab P 910591e01dfa488fabd5a14d0a764c3a: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-1/wal/wals/2caf66395f9245a9bd86e07fd94a66ab
I20260927 15:07:22.875128 8883 ts_tablet_manager.cc:1939] T 2caf66395f9245a9bd86e07fd94a66ab P 910591e01dfa488fabd5a14d0a764c3a: Deleting consensus metadata
I20260927 15:07:22.876423 8338 catalog_manager.cc:4841] TS 910591e01dfa488fabd5a14d0a764c3a (127.7.17.194:39493): tablet 2caf66395f9245a9bd86e07fd94a66ab (table test-table [id=5c8e3c72d6f14bf9a47fe7b4133214ad]) successfully deleted
I20260927 15:07:22.876971 8860 tablet_bootstrap.cc:492] T b270ae93a0064fdeb230978e02f8b07d P fc0870906b0f47b3acc65d63e1f9e04c: No bootstrap required, opened a new log
I20260927 15:07:22.877048 8860 ts_tablet_manager.cc:1397] T b270ae93a0064fdeb230978e02f8b07d P fc0870906b0f47b3acc65d63e1f9e04c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260927 15:07:22.877085 8861 tablet_bootstrap.cc:492] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7: Bootstrap starting.
I20260927 15:07:22.877480 8860 raft_consensus.cc:348] T b270ae93a0064fdeb230978e02f8b07d P fc0870906b0f47b3acc65d63e1f9e04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.877592 8860 raft_consensus.cc:374] T b270ae93a0064fdeb230978e02f8b07d P fc0870906b0f47b3acc65d63e1f9e04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:22.877620 8860 raft_consensus.cc:729] T b270ae93a0064fdeb230978e02f8b07d P fc0870906b0f47b3acc65d63e1f9e04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc0870906b0f47b3acc65d63e1f9e04c, State: Initialized, Role: FOLLOWER
I20260927 15:07:22.877729 8860 consensus_queue.cc:260] T b270ae93a0064fdeb230978e02f8b07d P fc0870906b0f47b3acc65d63e1f9e04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.877957 8860 ts_tablet_manager.cc:1428] T b270ae93a0064fdeb230978e02f8b07d P fc0870906b0f47b3acc65d63e1f9e04c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 15:07:22.877993 8862 tablet_bootstrap.cc:492] T b270ae93a0064fdeb230978e02f8b07d P 910591e01dfa488fabd5a14d0a764c3a: No bootstrap required, opened a new log
I20260927 15:07:22.878008 8861 tablet_bootstrap.cc:654] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:22.878068 8862 ts_tablet_manager.cc:1397] T b270ae93a0064fdeb230978e02f8b07d P 910591e01dfa488fabd5a14d0a764c3a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 15:07:22.878499 8862 raft_consensus.cc:348] T b270ae93a0064fdeb230978e02f8b07d P 910591e01dfa488fabd5a14d0a764c3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.878614 8862 raft_consensus.cc:374] T b270ae93a0064fdeb230978e02f8b07d P 910591e01dfa488fabd5a14d0a764c3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:22.878654 8862 raft_consensus.cc:729] T b270ae93a0064fdeb230978e02f8b07d P 910591e01dfa488fabd5a14d0a764c3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910591e01dfa488fabd5a14d0a764c3a, State: Initialized, Role: FOLLOWER
I20260927 15:07:22.878768 8862 consensus_queue.cc:260] T b270ae93a0064fdeb230978e02f8b07d P 910591e01dfa488fabd5a14d0a764c3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.879015 8862 ts_tablet_manager.cc:1428] T b270ae93a0064fdeb230978e02f8b07d P 910591e01dfa488fabd5a14d0a764c3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:22.880674 8861 tablet_bootstrap.cc:492] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7: No bootstrap required, opened a new log
I20260927 15:07:22.880775 8861 ts_tablet_manager.cc:1397] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 15:07:22.881260 8861 raft_consensus.cc:348] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.881393 8861 raft_consensus.cc:374] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:22.881422 8861 raft_consensus.cc:729] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7352f4752a784da297ffff12dfc133d7, State: Initialized, Role: FOLLOWER
I20260927 15:07:22.881549 8861 consensus_queue.cc:260] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.881819 8861 ts_tablet_manager.cc:1428] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 15:07:22.884783 8871 raft_consensus.cc:482] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:22.884950 8871 raft_consensus.cc:504] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.885339 8871 leader_election.cc:288] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 910591e01dfa488fabd5a14d0a764c3a (127.7.17.194:39493), fc0870906b0f47b3acc65d63e1f9e04c (127.7.17.195:35085)
I20260927 15:07:22.885711 8791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b270ae93a0064fdeb230978e02f8b07d" candidate_uuid: "7352f4752a784da297ffff12dfc133d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" is_pre_election: true
I20260927 15:07:22.885720 8651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b270ae93a0064fdeb230978e02f8b07d" candidate_uuid: "7352f4752a784da297ffff12dfc133d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "910591e01dfa488fabd5a14d0a764c3a" is_pre_election: true
I20260927 15:07:22.885842 8791 raft_consensus.cc:2454] T b270ae93a0064fdeb230978e02f8b07d P fc0870906b0f47b3acc65d63e1f9e04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7352f4752a784da297ffff12dfc133d7 in term 0.
I20260927 15:07:22.885859 8651 raft_consensus.cc:2454] T b270ae93a0064fdeb230978e02f8b07d P 910591e01dfa488fabd5a14d0a764c3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7352f4752a784da297ffff12dfc133d7 in term 0.
I20260927 15:07:22.886114 8442 leader_election.cc:302] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7352f4752a784da297ffff12dfc133d7, fc0870906b0f47b3acc65d63e1f9e04c; no voters:
I20260927 15:07:22.886266 8871 raft_consensus.cc:2781] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:22.886328 8871 raft_consensus.cc:482] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:22.886372 8871 raft_consensus.cc:3037] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:22.887290 8871 raft_consensus.cc:504] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.887562 8871 leader_election.cc:288] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7 [CANDIDATE]: Term 1 election: Requested vote from peers 910591e01dfa488fabd5a14d0a764c3a (127.7.17.194:39493), fc0870906b0f47b3acc65d63e1f9e04c (127.7.17.195:35085)
I20260927 15:07:22.887949 8791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b270ae93a0064fdeb230978e02f8b07d" candidate_uuid: "7352f4752a784da297ffff12dfc133d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc0870906b0f47b3acc65d63e1f9e04c"
I20260927 15:07:22.888060 8791 raft_consensus.cc:3037] T b270ae93a0064fdeb230978e02f8b07d P fc0870906b0f47b3acc65d63e1f9e04c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:22.888974 8651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b270ae93a0064fdeb230978e02f8b07d" candidate_uuid: "7352f4752a784da297ffff12dfc133d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "910591e01dfa488fabd5a14d0a764c3a"
I20260927 15:07:22.889086 8651 raft_consensus.cc:3037] T b270ae93a0064fdeb230978e02f8b07d P 910591e01dfa488fabd5a14d0a764c3a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:22.889117 8791 raft_consensus.cc:2454] T b270ae93a0064fdeb230978e02f8b07d P fc0870906b0f47b3acc65d63e1f9e04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7352f4752a784da297ffff12dfc133d7 in term 1.
I20260927 15:07:22.889393 8442 leader_election.cc:302] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7352f4752a784da297ffff12dfc133d7, fc0870906b0f47b3acc65d63e1f9e04c; no voters:
I20260927 15:07:22.889532 8871 raft_consensus.cc:2781] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:22.889601 8871 raft_consensus.cc:686] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7 [term 1 LEADER]: Becoming Leader. State: Replica: 7352f4752a784da297ffff12dfc133d7, State: Running, Role: LEADER
I20260927 15:07:22.889716 8871 consensus_queue.cc:237] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.890173 8651 raft_consensus.cc:2454] T b270ae93a0064fdeb230978e02f8b07d P 910591e01dfa488fabd5a14d0a764c3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7352f4752a784da297ffff12dfc133d7 in term 1.
I20260927 15:07:22.891187 8353 catalog_manager.cc:5495] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7352f4752a784da297ffff12dfc133d7 (127.7.17.193). New cstate: current_term: 1 leader_uuid: "7352f4752a784da297ffff12dfc133d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } health_report { overall_health: UNKNOWN } } }
Sep 27 15:07:22 dist-test-slave-hhpq krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521641, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:22.924017 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.914710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58250 (local address 127.7.17.254:43257)
0927 15:07:22.914822 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:22.914826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:22.915047 (+ 221us) server_negotiation.cc:407] Connection header received
0927 15:07:22.915169 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:22.915172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:22.915208 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:22.915271 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:22.915898 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.916438 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.917342 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.917622 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.917715 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.920188 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:22.920191 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:22.920576 (+ 385us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:22.920614 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:22.922279 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:22.922806 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:22.922810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:22.922812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:22.922888 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:22.923121 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:22.923125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:22.923127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:22.923319 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:22.923410 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:22.923700 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 15:07:22.923877 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":38}
I20260927 15:07:22.924019 8895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.914670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:43257 (local address 127.0.0.1:58250)
0927 15:07:22.914960 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:22.914973 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:22.915088 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:22.915298 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:22.915301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:22.915703 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:22.915794 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.915800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.916533 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.916536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.917235 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.917245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.917723 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.917727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.917796 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.918414 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:22.918426 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:22.920045 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:22.922416 (+ 2371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:22.922432 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:22.922440 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:22.922713 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:22.922968 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:22.922971 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:22.922973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:22.923047 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:22.923422 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:22.923431 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:22.923545 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 15:07:22.923641 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 15:07:22.923812 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":109,"threads_started":1}
I20260927 15:07:22.929374 8353 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:58250:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 15:07:22.930133 8353 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:22.935550 8491 tablet_service.cc:1467] Processing CreateTablet for tablet da3dd0bb44fb4419a39ee532da6eaec9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac9c9db650a941169e209eee838e4c0a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:07:22.935550 8631 tablet_service.cc:1467] Processing CreateTablet for tablet da3dd0bb44fb4419a39ee532da6eaec9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac9c9db650a941169e209eee838e4c0a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:07:22.935931 8491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da3dd0bb44fb4419a39ee532da6eaec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:22.935935 8631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da3dd0bb44fb4419a39ee532da6eaec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:22.938350 8861 tablet_bootstrap.cc:492] T da3dd0bb44fb4419a39ee532da6eaec9 P 7352f4752a784da297ffff12dfc133d7: Bootstrap starting.
I20260927 15:07:22.938730 8771 tablet_service.cc:1467] Processing CreateTablet for tablet da3dd0bb44fb4419a39ee532da6eaec9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac9c9db650a941169e209eee838e4c0a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:07:22.938900 8862 tablet_bootstrap.cc:492] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a: Bootstrap starting.
I20260927 15:07:22.939113 8771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da3dd0bb44fb4419a39ee532da6eaec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:22.939345 8861 tablet_bootstrap.cc:654] T da3dd0bb44fb4419a39ee532da6eaec9 P 7352f4752a784da297ffff12dfc133d7: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:22.939903 8862 tablet_bootstrap.cc:654] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:22.941295 8862 tablet_bootstrap.cc:492] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a: No bootstrap required, opened a new log
I20260927 15:07:22.941390 8862 ts_tablet_manager.cc:1397] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 15:07:22.941823 8862 raft_consensus.cc:348] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.941941 8862 raft_consensus.cc:374] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:22.941968 8862 raft_consensus.cc:729] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910591e01dfa488fabd5a14d0a764c3a, State: Initialized, Role: FOLLOWER
I20260927 15:07:22.941982 8860 tablet_bootstrap.cc:492] T da3dd0bb44fb4419a39ee532da6eaec9 P fc0870906b0f47b3acc65d63e1f9e04c: Bootstrap starting.
I20260927 15:07:22.942085 8862 consensus_queue.cc:260] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.942343 8862 ts_tablet_manager.cc:1428] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:22.942983 8860 tablet_bootstrap.cc:654] T da3dd0bb44fb4419a39ee532da6eaec9 P fc0870906b0f47b3acc65d63e1f9e04c: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:22.942973 8868 tablet_replica.cc:439] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } } }
I20260927 15:07:22.943114 8868 tablet_replica.cc:442] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:22.943922 8861 tablet_bootstrap.cc:492] T da3dd0bb44fb4419a39ee532da6eaec9 P 7352f4752a784da297ffff12dfc133d7: No bootstrap required, opened a new log
I20260927 15:07:22.944010 8861 ts_tablet_manager.cc:1397] T da3dd0bb44fb4419a39ee532da6eaec9 P 7352f4752a784da297ffff12dfc133d7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 15:07:22.944403 8861 raft_consensus.cc:348] T da3dd0bb44fb4419a39ee532da6eaec9 P 7352f4752a784da297ffff12dfc133d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.944528 8861 raft_consensus.cc:374] T da3dd0bb44fb4419a39ee532da6eaec9 P 7352f4752a784da297ffff12dfc133d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:22.944561 8861 raft_consensus.cc:729] T da3dd0bb44fb4419a39ee532da6eaec9 P 7352f4752a784da297ffff12dfc133d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7352f4752a784da297ffff12dfc133d7, State: Initialized, Role: FOLLOWER
I20260927 15:07:22.944875 8861 consensus_queue.cc:260] T da3dd0bb44fb4419a39ee532da6eaec9 P 7352f4752a784da297ffff12dfc133d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.945147 8861 ts_tablet_manager.cc:1428] T da3dd0bb44fb4419a39ee532da6eaec9 P 7352f4752a784da297ffff12dfc133d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:22.945381 8871 tablet_replica.cc:439] T da3dd0bb44fb4419a39ee532da6eaec9 P 7352f4752a784da297ffff12dfc133d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } } }
I20260927 15:07:22.945497 8871 tablet_replica.cc:442] T da3dd0bb44fb4419a39ee532da6eaec9 P 7352f4752a784da297ffff12dfc133d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:22.946363 8860 tablet_bootstrap.cc:492] T da3dd0bb44fb4419a39ee532da6eaec9 P fc0870906b0f47b3acc65d63e1f9e04c: No bootstrap required, opened a new log
I20260927 15:07:22.946467 8860 ts_tablet_manager.cc:1397] T da3dd0bb44fb4419a39ee532da6eaec9 P fc0870906b0f47b3acc65d63e1f9e04c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260927 15:07:22.946928 8860 raft_consensus.cc:348] T da3dd0bb44fb4419a39ee532da6eaec9 P fc0870906b0f47b3acc65d63e1f9e04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.947045 8860 raft_consensus.cc:374] T da3dd0bb44fb4419a39ee532da6eaec9 P fc0870906b0f47b3acc65d63e1f9e04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:22.947083 8860 raft_consensus.cc:729] T da3dd0bb44fb4419a39ee532da6eaec9 P fc0870906b0f47b3acc65d63e1f9e04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc0870906b0f47b3acc65d63e1f9e04c, State: Initialized, Role: FOLLOWER
I20260927 15:07:22.947197 8860 consensus_queue.cc:260] T da3dd0bb44fb4419a39ee532da6eaec9 P fc0870906b0f47b3acc65d63e1f9e04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.947443 8860 ts_tablet_manager.cc:1428] T da3dd0bb44fb4419a39ee532da6eaec9 P fc0870906b0f47b3acc65d63e1f9e04c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 15:07:22.947456 8867 tablet_replica.cc:439] T da3dd0bb44fb4419a39ee532da6eaec9 P fc0870906b0f47b3acc65d63e1f9e04c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } } }
I20260927 15:07:22.947703 8867 tablet_replica.cc:442] T da3dd0bb44fb4419a39ee532da6eaec9 P fc0870906b0f47b3acc65d63e1f9e04c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:22.955878 8868 raft_consensus.cc:482] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:22.956010 8868 raft_consensus.cc:504] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.957223 8868 leader_election.cc:288] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7352f4752a784da297ffff12dfc133d7 (127.7.17.193:37311), fc0870906b0f47b3acc65d63e1f9e04c (127.7.17.195:35085)
I20260927 15:07:22.962258 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.957824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:53715 (local address 127.7.17.193:37311)
0927 15:07:22.957952 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:22.957957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:22.957973 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:07:22.958090 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:22.958094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:22.958136 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:22.958179 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:22.958674 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.959452 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.961286 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.961677 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.961794 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.961855 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:22.962039 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 15:07:22.962124 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:07:22.962437 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.957451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:35085 (local address 127.7.17.194:33705)
0927 15:07:22.957841 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:22.957855 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:22.957976 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:22.958310 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:22.958313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:22.958341 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:22.958547 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.958553 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.959600 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.959603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.961123 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.961136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.962033 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.962037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.962129 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.962150 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:07:22.962218 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 15:07:22.962291 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":133,"spinlock_wait_cycles":8320,"thread_start_us":116,"threads_started":1}
I20260927 15:07:22.962467 8859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.957597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:33705 (local address 127.7.17.195:35085)
0927 15:07:22.957733 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:22.957737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:22.957928 (+ 191us) server_negotiation.cc:407] Connection header received
0927 15:07:22.958085 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:22.958088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:22.958133 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:22.958179 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:22.958675 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.959456 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.961263 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.961906 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.962028 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.962078 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:22.962244 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 15:07:22.962331 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:07:22.962975 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.957431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:37311 (local address 127.7.17.194:53715)
0927 15:07:22.957841 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:22.957855 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:22.957962 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:22.958309 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:22.958312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:22.958335 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:22.958543 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.958551 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.959600 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.959603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.961123 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.961136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.961795 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.961798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.961901 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.961924 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:07:22.962747 (+ 823us) client_negotiation.cc:240] Negotiation successful
0927 15:07:22.962834 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":133,"spinlock_wait_cycles":768,"thread_start_us":130,"threads_started":1}
I20260927 15:07:22.962946 8791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da3dd0bb44fb4419a39ee532da6eaec9" candidate_uuid: "910591e01dfa488fabd5a14d0a764c3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" is_pre_election: true
I20260927 15:07:22.963104 8791 raft_consensus.cc:2454] T da3dd0bb44fb4419a39ee532da6eaec9 P fc0870906b0f47b3acc65d63e1f9e04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910591e01dfa488fabd5a14d0a764c3a in term 0.
I20260927 15:07:22.963501 8511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da3dd0bb44fb4419a39ee532da6eaec9" candidate_uuid: "910591e01dfa488fabd5a14d0a764c3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7352f4752a784da297ffff12dfc133d7" is_pre_election: true
I20260927 15:07:22.963526 8582 leader_election.cc:302] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910591e01dfa488fabd5a14d0a764c3a, fc0870906b0f47b3acc65d63e1f9e04c; no voters:
I20260927 15:07:22.963730 8511 raft_consensus.cc:2454] T da3dd0bb44fb4419a39ee532da6eaec9 P 7352f4752a784da297ffff12dfc133d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910591e01dfa488fabd5a14d0a764c3a in term 0.
I20260927 15:07:22.963791 8868 raft_consensus.cc:2781] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:22.963856 8868 raft_consensus.cc:482] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:22.963901 8868 raft_consensus.cc:3037] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:22.965159 8868 raft_consensus.cc:504] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.965544 8868 leader_election.cc:288] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a [CANDIDATE]: Term 1 election: Requested vote from peers 7352f4752a784da297ffff12dfc133d7 (127.7.17.193:37311), fc0870906b0f47b3acc65d63e1f9e04c (127.7.17.195:35085)
I20260927 15:07:22.965865 8791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da3dd0bb44fb4419a39ee532da6eaec9" candidate_uuid: "910591e01dfa488fabd5a14d0a764c3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc0870906b0f47b3acc65d63e1f9e04c"
I20260927 15:07:22.965915 8511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da3dd0bb44fb4419a39ee532da6eaec9" candidate_uuid: "910591e01dfa488fabd5a14d0a764c3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7352f4752a784da297ffff12dfc133d7"
I20260927 15:07:22.965994 8791 raft_consensus.cc:3037] T da3dd0bb44fb4419a39ee532da6eaec9 P fc0870906b0f47b3acc65d63e1f9e04c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:22.966025 8511 raft_consensus.cc:3037] T da3dd0bb44fb4419a39ee532da6eaec9 P 7352f4752a784da297ffff12dfc133d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:22.967016 8791 raft_consensus.cc:2454] T da3dd0bb44fb4419a39ee532da6eaec9 P fc0870906b0f47b3acc65d63e1f9e04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910591e01dfa488fabd5a14d0a764c3a in term 1.
I20260927 15:07:22.967192 8511 raft_consensus.cc:2454] T da3dd0bb44fb4419a39ee532da6eaec9 P 7352f4752a784da297ffff12dfc133d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910591e01dfa488fabd5a14d0a764c3a in term 1.
I20260927 15:07:22.967339 8582 leader_election.cc:302] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910591e01dfa488fabd5a14d0a764c3a, fc0870906b0f47b3acc65d63e1f9e04c; no voters:
I20260927 15:07:22.967506 8868 raft_consensus.cc:2781] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:22.967715 8868 raft_consensus.cc:686] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a [term 1 LEADER]: Becoming Leader. State: Replica: 910591e01dfa488fabd5a14d0a764c3a, State: Running, Role: LEADER
I20260927 15:07:22.967938 8868 consensus_queue.cc:237] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } }
I20260927 15:07:22.969837 8901 tablet_replica.cc:439] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a: TxnStatusTablet state changed. Reason: New leader 910591e01dfa488fabd5a14d0a764c3a. Latest consensus state: current_term: 1 leader_uuid: "910591e01dfa488fabd5a14d0a764c3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } } }
I20260927 15:07:22.969944 8901 tablet_replica.cc:442] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:07:22.970198 8903 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 15:07:22.970329 8903 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 15:07:22.970465 8353 catalog_manager.cc:5495] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 910591e01dfa488fabd5a14d0a764c3a (127.7.17.194). New cstate: current_term: 1 leader_uuid: "910591e01dfa488fabd5a14d0a764c3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } health_report { overall_health: UNKNOWN } } }
I20260927 15:07:22.987584 8856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.983918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51602 (local address 127.7.17.194:39493)
0927 15:07:22.984033 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:22.984037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:22.984051 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:07:22.984265 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:22.984269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:22.984334 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:22.984418 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:22.984646 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.985216 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.986130 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.986357 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.986443 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.986752 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:22.987177 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:22.987279 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:22.987375 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 15:07:22.987435 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:07:22.987584 8895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:22.983825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:39493 (local address 127.0.0.1:51602)
0927 15:07:22.983990 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:22.984007 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:22.984124 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:22.984439 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:22.984442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:22.984453 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:07:22.984549 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.984554 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.985318 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.985321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.986020 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:22.986029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:22.986455 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:22.986459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:22.986543 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:22.986590 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:07:22.987270 (+ 680us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:07:22.987294 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:07:22.987358 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 15:07:22.987447 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 15:07:22.990622 8791 raft_consensus.cc:1261] T da3dd0bb44fb4419a39ee532da6eaec9 P fc0870906b0f47b3acc65d63e1f9e04c [term 1 FOLLOWER]: Refusing update from remote peer 910591e01dfa488fabd5a14d0a764c3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:22.990821 8511 raft_consensus.cc:1261] T da3dd0bb44fb4419a39ee532da6eaec9 P 7352f4752a784da297ffff12dfc133d7 [term 1 FOLLOWER]: Refusing update from remote peer 910591e01dfa488fabd5a14d0a764c3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:22.991358 8868 consensus_queue.cc:1035] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:22.991729 8868 consensus_queue.cc:1035] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:22.993234 8867 tablet_replica.cc:439] T da3dd0bb44fb4419a39ee532da6eaec9 P fc0870906b0f47b3acc65d63e1f9e04c: TxnStatusTablet state changed. Reason: New leader 910591e01dfa488fabd5a14d0a764c3a. Latest consensus state: current_term: 1 leader_uuid: "910591e01dfa488fabd5a14d0a764c3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } } }
I20260927 15:07:22.993360 8867 tablet_replica.cc:442] T da3dd0bb44fb4419a39ee532da6eaec9 P fc0870906b0f47b3acc65d63e1f9e04c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:22.993319 8871 tablet_replica.cc:439] T da3dd0bb44fb4419a39ee532da6eaec9 P 7352f4752a784da297ffff12dfc133d7: TxnStatusTablet state changed. Reason: New leader 910591e01dfa488fabd5a14d0a764c3a. Latest consensus state: current_term: 1 leader_uuid: "910591e01dfa488fabd5a14d0a764c3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } } }
I20260927 15:07:22.993427 8871 tablet_replica.cc:442] T da3dd0bb44fb4419a39ee532da6eaec9 P 7352f4752a784da297ffff12dfc133d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:22.995893 8868 tablet_replica.cc:439] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "910591e01dfa488fabd5a14d0a764c3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } } }
I20260927 15:07:22.996027 8868 tablet_replica.cc:442] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:07:22.996621 8871 tablet_replica.cc:439] T da3dd0bb44fb4419a39ee532da6eaec9 P 7352f4752a784da297ffff12dfc133d7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "910591e01dfa488fabd5a14d0a764c3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } } }
I20260927 15:07:22.996730 8871 tablet_replica.cc:442] T da3dd0bb44fb4419a39ee532da6eaec9 P 7352f4752a784da297ffff12dfc133d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:23.003408 8901 tablet_replica.cc:439] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "910591e01dfa488fabd5a14d0a764c3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } } }
I20260927 15:07:23.003549 8901 tablet_replica.cc:442] T da3dd0bb44fb4419a39ee532da6eaec9 P 910591e01dfa488fabd5a14d0a764c3a: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:07:23.004901 8876 mvcc.cc:205] Tried to move back new op lower bound from 7333976649684422656 to 7333976649599205376. Current Snapshot: MvccSnapshot[applied={T|T < 7333976649684422656}]
I20260927 15:07:23.005100 8867 tablet_replica.cc:439] T da3dd0bb44fb4419a39ee532da6eaec9 P fc0870906b0f47b3acc65d63e1f9e04c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "910591e01dfa488fabd5a14d0a764c3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7352f4752a784da297ffff12dfc133d7" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37311 } } peers { permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 } } peers { permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 } } }
I20260927 15:07:23.005203 8867 tablet_replica.cc:442] T da3dd0bb44fb4419a39ee532da6eaec9 P fc0870906b0f47b3acc65d63e1f9e04c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:23.022966 8856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.018283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:33873 (local address 127.7.17.194:39493)
0927 15:07:23.018404 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:23.018409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:23.018723 (+ 314us) server_negotiation.cc:407] Connection header received
0927 15:07:23.018937 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:23.018940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:23.018992 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:23.019037 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:23.019404 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.020196 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.021590 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.021947 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.022021 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.022457 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:23.022553 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:23.022595 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:23.022741 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 15:07:23.022813 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:07:23.023492 8910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.017891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:39493 (local address 127.7.17.193:33873)
0927 15:07:23.018638 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:23.018652 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:23.018818 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:23.019162 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:23.019165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:23.019172 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:07:23.019289 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:23.019295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.020343 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.020347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:23.021439 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:23.021453 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.022139 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.022143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:23.022245 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.022304 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:07:23.022651 (+ 347us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:07:23.022656 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:07:23.023243 (+ 587us) client_negotiation.cc:240] Negotiation successful
0927 15:07:23.023338 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":114,"threads_started":1}
I20260927 15:07:23.032459 8791 raft_consensus.cc:1261] T b270ae93a0064fdeb230978e02f8b07d P fc0870906b0f47b3acc65d63e1f9e04c [term 1 FOLLOWER]: Refusing update from remote peer 7352f4752a784da297ffff12dfc133d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:23.032495 8651 raft_consensus.cc:1261] T b270ae93a0064fdeb230978e02f8b07d P 910591e01dfa488fabd5a14d0a764c3a [term 1 FOLLOWER]: Refusing update from remote peer 7352f4752a784da297ffff12dfc133d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:23.032968 8871 consensus_queue.cc:1035] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910591e01dfa488fabd5a14d0a764c3a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:23.033257 8871 consensus_queue.cc:1035] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc0870906b0f47b3acc65d63e1f9e04c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:23.060179 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.056429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:60491 (local address 127.7.17.193:37311)
0927 15:07:23.056550 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:23.056554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:23.056717 (+ 163us) server_negotiation.cc:407] Connection header received
0927 15:07:23.056850 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:23.056853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:23.056909 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:23.056991 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:23.057188 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.057983 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.059070 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.059320 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.059411 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.059740 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:23.059821 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:23.059879 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:23.059971 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 15:07:23.060028 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:07:23.060194 8915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.056341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:37311 (local address 127.7.17.194:60491)
0927 15:07:23.056645 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:23.056658 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:23.056747 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:23.056995 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:23.056997 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:23.057002 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:07:23.057107 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:23.057111 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.058117 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.058120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:23.058960 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:23.058973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.059428 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.059432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:23.059527 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.059579 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:07:23.059899 (+ 320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:07:23.059906 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:07:23.059961 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 15:07:23.060041 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":101,"threads_started":1}
I20260927 15:07:23.082461 8354 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:23.082625 8354 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:23.084568 8354 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80af636a3cec4b16b03eb336950e2901: Sending DeleteTablet for 3 replicas of tablet b270ae93a0064fdeb230978e02f8b07d
I20260927 15:07:23.085117 8491 tablet_service.cc:1514] Processing DeleteTablet for tablet b270ae93a0064fdeb230978e02f8b07d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:23 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:34576
I20260927 15:07:23.085216 8771 tablet_service.cc:1514] Processing DeleteTablet for tablet b270ae93a0064fdeb230978e02f8b07d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:23 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:56444
I20260927 15:07:23.085215 8631 tablet_service.cc:1514] Processing DeleteTablet for tablet b270ae93a0064fdeb230978e02f8b07d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:23 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:51590
I20260927 15:07:23.085307 8884 tablet_replica.cc:330] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7: stopping tablet replica
I20260927 15:07:23.085392 8883 tablet_replica.cc:330] T b270ae93a0064fdeb230978e02f8b07d P 910591e01dfa488fabd5a14d0a764c3a: stopping tablet replica
I20260927 15:07:23.085412 8884 raft_consensus.cc:2229] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:07:23.085482 8883 raft_consensus.cc:2229] T b270ae93a0064fdeb230978e02f8b07d P 910591e01dfa488fabd5a14d0a764c3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:23.085572 8883 raft_consensus.cc:2258] T b270ae93a0064fdeb230978e02f8b07d P 910591e01dfa488fabd5a14d0a764c3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:23.085593 8884 raft_consensus.cc:2258] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:23.085759 8885 tablet_replica.cc:330] T b270ae93a0064fdeb230978e02f8b07d P fc0870906b0f47b3acc65d63e1f9e04c: stopping tablet replica
I20260927 15:07:23.085868 8885 raft_consensus.cc:2229] T b270ae93a0064fdeb230978e02f8b07d P fc0870906b0f47b3acc65d63e1f9e04c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:23.085970 8885 raft_consensus.cc:2258] T b270ae93a0064fdeb230978e02f8b07d P fc0870906b0f47b3acc65d63e1f9e04c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:23.086422 8883 ts_tablet_manager.cc:1905] T b270ae93a0064fdeb230978e02f8b07d P 910591e01dfa488fabd5a14d0a764c3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:23.086796 8885 ts_tablet_manager.cc:1905] T b270ae93a0064fdeb230978e02f8b07d P fc0870906b0f47b3acc65d63e1f9e04c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:23.088678 8883 ts_tablet_manager.cc:1918] T b270ae93a0064fdeb230978e02f8b07d P 910591e01dfa488fabd5a14d0a764c3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:07:23.088749 8883 log.cc:1196] T b270ae93a0064fdeb230978e02f8b07d P 910591e01dfa488fabd5a14d0a764c3a: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-1/wal/wals/b270ae93a0064fdeb230978e02f8b07d
I20260927 15:07:23.089296 8884 ts_tablet_manager.cc:1905] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:23.089809 8885 ts_tablet_manager.cc:1918] T b270ae93a0064fdeb230978e02f8b07d P fc0870906b0f47b3acc65d63e1f9e04c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:07:23.089883 8885 log.cc:1196] T b270ae93a0064fdeb230978e02f8b07d P fc0870906b0f47b3acc65d63e1f9e04c: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/b270ae93a0064fdeb230978e02f8b07d
I20260927 15:07:23.090119 8885 ts_tablet_manager.cc:1939] T b270ae93a0064fdeb230978e02f8b07d P fc0870906b0f47b3acc65d63e1f9e04c: Deleting consensus metadata
I20260927 15:07:23.090898 8338 catalog_manager.cc:4841] TS fc0870906b0f47b3acc65d63e1f9e04c (127.7.17.195:35085): tablet b270ae93a0064fdeb230978e02f8b07d (table test-table [id=98b7a1beba9546338faa27a4229f5c15]) successfully deleted
I20260927 15:07:23.091934 8884 ts_tablet_manager.cc:1918] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:07:23.091990 8884 log.cc:1196] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521636012660-7239-0/minicluster-data/ts-0/wal/wals/b270ae93a0064fdeb230978e02f8b07d
I20260927 15:07:23.092145 8884 ts_tablet_manager.cc:1939] T b270ae93a0064fdeb230978e02f8b07d P 7352f4752a784da297ffff12dfc133d7: Deleting consensus metadata
I20260927 15:07:23.092446 8883 ts_tablet_manager.cc:1939] T b270ae93a0064fdeb230978e02f8b07d P 910591e01dfa488fabd5a14d0a764c3a: Deleting consensus metadata
I20260927 15:07:23.093029 8340 catalog_manager.cc:4841] TS 7352f4752a784da297ffff12dfc133d7 (127.7.17.193:37311): tablet b270ae93a0064fdeb230978e02f8b07d (table test-table [id=98b7a1beba9546338faa27a4229f5c15]) successfully deleted
I20260927 15:07:23.093147 8338 catalog_manager.cc:4841] TS 910591e01dfa488fabd5a14d0a764c3a (127.7.17.194:39493): tablet b270ae93a0064fdeb230978e02f8b07d (table test-table [id=98b7a1beba9546338faa27a4229f5c15]) successfully deleted
Sep 27 15:07:23 dist-test-slave-hhpq krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521642, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.17.193@KRBTEST.COM
I20260927 15:07:23.102885 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.093465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34598 (local address 127.7.17.193:37311)
0927 15:07:23.093584 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:23.093589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:23.093873 (+ 284us) server_negotiation.cc:407] Connection header received
0927 15:07:23.094000 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:23.094002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:23.094051 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:23.094119 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:23.094858 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.095434 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.096274 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.096554 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.096642 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.099579 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:23.099593 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:23.099608 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:23.099633 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:23.101213 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:23.101630 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:23.101636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:23.101637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:23.101714 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:23.101957 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:23.101960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:23.101962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:23.102210 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:23.102426 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:23.102632 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 15:07:23.102750 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 15:07:23.103649 8548 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34598
I20260927 15:07:23.113072 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:23.105380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58252 (local address 127.7.17.254:43257)
0927 15:07:23.105514 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:23.105519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:23.105533 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:07:23.105636 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:23.105639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:23.105688 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:23.105769 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:23.106329 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.107129 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.107976 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:23.108333 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:23.108442 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:23.109874 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:23.109878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:23.109880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:23.109911 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:23.111433 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:23.111824 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:23.111828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:23.111829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:23.111924 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:23.112184 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:23.112188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:23.112190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:23.112434 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:23.112560 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:23.112755 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 15:07:23.112913 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 15:07:23.113530 8354 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58252
I20260927 15:07:23.114732 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 8426
I20260927 15:07:23.122229 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 8566
I20260927 15:07:23.128132 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 8706
I20260927 15:07:23.133188 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 8322
2026-09-27T15:07:23Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2675 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:07:23 dist-test-slave-hhpq krb5kdc[8923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:07:23 dist-test-slave-hhpq krb5kdc[8923](info): set up 2 sockets
Sep 27 15:07:23 dist-test-slave-hhpq krb5kdc[8923](info): commencing operation
krb5kdc: starting...
W20260927 15:07:24.100495 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:07:24 dist-test-slave-hhpq krb5kdc[8923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:07:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:07:24.223547 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:40013
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:46797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:40013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:24.335765 8936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:24.336068 8936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:24.336126 8936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:24.340220 8936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:24.340286 8936 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 15:07:24.340308 8936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:24.340332 8936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:24.340354 8936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:24.340384 8936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:24.345903 8936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:46797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:40013
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:40013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:24.347033 8936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:24.348078 8936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:24.356693 8942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:24.356703 8941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:24.356755 8936 server_base.cc:1034] running on GCE node
W20260927 15:07:24.356694 8944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:24.357518 8936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:24.358657 8936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:24.359833 8936 hybrid_clock.cc:648] HybridClock initialized: now 1790521644359816 us; error 29 us; skew 500 ppm
Sep 27 15:07:24 dist-test-slave-hhpq krb5kdc[8923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521644, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:24.362598 8936 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:07:24.363547 8936 webserver.cc:460] Webserver started at http://127.7.17.254:39315/ using document root <none> and password file <none>
I20260927 15:07:24.364110 8936 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:24.364176 8936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:24.364375 8936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:24.366503 8936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "136271aab3344dc0b0ae2ff9f45bd0d1"
format_stamp: "Formatted at 2026-09-27 15:07:24 on dist-test-slave-hhpq"
server_key: "a5e5504354f6f6e1516666d0f4c8560b"
server_key_iv: "61d5bb44c40f1c2a94c6659fe89a92be"
server_key_version: "encryptionkey@0"
I20260927 15:07:24.367074 8936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "136271aab3344dc0b0ae2ff9f45bd0d1"
format_stamp: "Formatted at 2026-09-27 15:07:24 on dist-test-slave-hhpq"
server_key: "a5e5504354f6f6e1516666d0f4c8560b"
server_key_iv: "61d5bb44c40f1c2a94c6659fe89a92be"
server_key_version: "encryptionkey@0"
I20260927 15:07:24.371008 8936 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:07:24.373652 8951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:24.375008 8936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:24.375140 8936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "136271aab3344dc0b0ae2ff9f45bd0d1"
format_stamp: "Formatted at 2026-09-27 15:07:24 on dist-test-slave-hhpq"
server_key: "a5e5504354f6f6e1516666d0f4c8560b"
server_key_iv: "61d5bb44c40f1c2a94c6659fe89a92be"
server_key_version: "encryptionkey@0"
I20260927 15:07:24.375257 8936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:24.387976 8936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:24.391327 8936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:24.391510 8936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:24.399174 8936 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:40013
I20260927 15:07:24.399205 9013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:40013 every 8 connection(s)
I20260927 15:07:24.400141 8936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:24.403014 9014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:24.409960 9014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1: Bootstrap starting.
I20260927 15:07:24.410845 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 8936
I20260927 15:07:24.410960 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:07:24.411250 7239 external_mini_cluster.cc:1428] Setting key 8fcf7a697edcdccb7b4c4cfadee27c21
I20260927 15:07:24.413753 9014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:24.414752 9014 log.cc:824] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:24.417240 9014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1: No bootstrap required, opened a new log
I20260927 15:07:24.422360 9014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136271aab3344dc0b0ae2ff9f45bd0d1" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 40013 } }
I20260927 15:07:24.422701 9014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:24.422765 9014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136271aab3344dc0b0ae2ff9f45bd0d1, State: Initialized, Role: FOLLOWER
I20260927 15:07:24.423338 9014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136271aab3344dc0b0ae2ff9f45bd0d1" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 40013 } }
I20260927 15:07:24.423516 9014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 27 15:07:24 dist-test-slave-hhpq krb5kdc[8923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:24.423586 9014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:24.424495 9014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:24.425798 9014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136271aab3344dc0b0ae2ff9f45bd0d1" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 40013 } }
I20260927 15:07:24.426204 9014 leader_election.cc:302] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 136271aab3344dc0b0ae2ff9f45bd0d1; no voters:
I20260927 15:07:24.426558 9014 leader_election.cc:288] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:24.426652 9019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:24.426816 9019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1 [term 1 LEADER]: Becoming Leader. State: Replica: 136271aab3344dc0b0ae2ff9f45bd0d1, State: Running, Role: LEADER
I20260927 15:07:24.427139 9019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136271aab3344dc0b0ae2ff9f45bd0d1" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 40013 } }
I20260927 15:07:24.427943 9014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:24.428400 9019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 136271aab3344dc0b0ae2ff9f45bd0d1. Latest consensus state: current_term: 1 leader_uuid: "136271aab3344dc0b0ae2ff9f45bd0d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136271aab3344dc0b0ae2ff9f45bd0d1" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 40013 } } }
I20260927 15:07:24.428525 9019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:24.428938 9023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:24.428952 9020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "136271aab3344dc0b0ae2ff9f45bd0d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136271aab3344dc0b0ae2ff9f45bd0d1" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 40013 } } }
I20260927 15:07:24.429899 9020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:24.431056 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.413891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51476 (local address 127.7.17.254:40013)
0927 15:07:24.414520 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:24.414541 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:24.414589 (+ 48us) server_negotiation.cc:407] Connection header received
0927 15:07:24.415528 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:24.415557 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:24.416149 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:24.416547 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:24.418096 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.419665 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.421096 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.421878 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.422056 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.425098 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:24.425106 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:24.425126 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:24.425160 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:24.427153 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:24.427688 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:24.427694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:24.427701 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:24.427850 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:24.428402 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:24.428406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:24.428408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:24.429065 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:24.429289 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:24.429621 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 15:07:24.429991 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":163,"threads_started":1}
I20260927 15:07:24.434474 9023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:24.441893 9023 catalog_manager.cc:1347] Generated new cluster ID: dc84a4366dbc4c53aeea062cabc697dd
I20260927 15:07:24.441988 9023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:24.458951 9023 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:24.460307 9023 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:24.469611 9023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1: Generated new TSK 0
I20260927 15:07:24.470367 9023 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
I20260927 15:07:24.502126 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:40013
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:46797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:24.614641 9040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:24.614952 9040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:24.615020 9040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:24.619200 9040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:24.619280 9040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:24.619338 9040 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 15:07:24.619364 9040 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:07:24.619428 9040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:07:24.625018 9040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:46797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.17.254:40013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:24.626155 9040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:24.627220 9040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:24.637403 9046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:24.637435 9048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:24.637405 9045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:24.637620 9040 server_base.cc:1034] running on GCE node
I20260927 15:07:24.638204 9040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:24.638875 9040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:24.640050 9040 hybrid_clock.cc:648] HybridClock initialized: now 1790521644640021 us; error 42 us; skew 500 ppm
Sep 27 15:07:24 dist-test-slave-hhpq krb5kdc[8923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521644, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:24.643049 9040 init.cc:377] Logged in from keytab as kudu/127.7.17.193@KRBTEST.COM (short username kudu)
I20260927 15:07:24.644009 9040 webserver.cc:460] Webserver started at http://127.7.17.193:36097/ using document root <none> and password file <none>
I20260927 15:07:24.644641 9040 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:24.644707 9040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:24.644939 9040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:24.647172 9040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "e676da6006f24776bc62de5cd993cee5"
format_stamp: "Formatted at 2026-09-27 15:07:24 on dist-test-slave-hhpq"
server_key: "1cb2940c2dd6a450cec8f3560e2edabf"
server_key_iv: "bb7fa21e17affd25d9cf4a6030fb568f"
server_key_version: "encryptionkey@0"
I20260927 15:07:24.647765 9040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "e676da6006f24776bc62de5cd993cee5"
format_stamp: "Formatted at 2026-09-27 15:07:24 on dist-test-slave-hhpq"
server_key: "1cb2940c2dd6a450cec8f3560e2edabf"
server_key_iv: "bb7fa21e17affd25d9cf4a6030fb568f"
server_key_version: "encryptionkey@0"
I20260927 15:07:24.651930 9040 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:07:24.654590 9055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:24.655895 9040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:07:24.655999 9040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "e676da6006f24776bc62de5cd993cee5"
format_stamp: "Formatted at 2026-09-27 15:07:24 on dist-test-slave-hhpq"
server_key: "1cb2940c2dd6a450cec8f3560e2edabf"
server_key_iv: "bb7fa21e17affd25d9cf4a6030fb568f"
server_key_version: "encryptionkey@0"
I20260927 15:07:24.656114 9040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:24.673877 9040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:24.677101 9040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:24.677280 9040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:24.678879 9040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:24.678954 9040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:24.679013 9040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:24.679039 9040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:07:24 dist-test-slave-hhpq krb5kdc[8923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521644, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:24.689688 9040 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:33045
I20260927 15:07:24.689716 9170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:33045 every 8 connection(s)
I20260927 15:07:24.690856 9040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:24.693521 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.679686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:57717 (local address 127.7.17.254:40013)
0927 15:07:24.679803 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:24.679807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:24.681035 (+ 1228us) server_negotiation.cc:407] Connection header received
0927 15:07:24.682179 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:24.682184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:24.682253 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:24.682361 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:24.683651 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.684478 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.685917 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.686165 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.686264 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.689582 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:24.689587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:24.689590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:24.689625 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:24.691258 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:24.691873 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:24.691877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:24.691879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:24.691978 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:24.692295 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:24.692358 (+ 63us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:24.692360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:24.692541 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:24.692616 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:24.693258 (+ 642us) server_negotiation.cc:299] Negotiation successful
0927 15:07:24.693382 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:07:24.694330 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.680338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:40013 (local address 127.7.17.193:57717)
0927 15:07:24.680888 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:24.680935 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:24.681883 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:24.682554 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:24.682563 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:24.682950 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:24.683498 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.683514 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.684627 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.684631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.685788 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.685799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.686327 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.686331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.686525 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.687694 (+ 1169us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:24.687718 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:24.689326 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:24.691400 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:24.691406 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:24.691423 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:24.691744 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:24.692073 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:24.692076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:24.692078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:24.692219 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:24.692629 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:24.692634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:24.692939 (+ 305us) client_negotiation.cc:769] Sending connection context
0927 15:07:24.693191 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 15:07:24.693472 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":138,"threads_started":1}
I20260927 15:07:24.695576 9171 heartbeater.cc:347] Connected to a master server at 127.7.17.254:40013
I20260927 15:07:24.695716 9171 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:24.696274 9171 heartbeater.cc:511] Master 127.7.17.254:40013 requested a full tablet report, sending...
I20260927 15:07:24.698014 8968 ts_manager.cc:194] Registered new tserver with Master: e676da6006f24776bc62de5cd993cee5 (127.7.17.193:33045)
I20260927 15:07:24.699084 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 9040
I20260927 15:07:24.699205 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:07:24.699499 8968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:57717
I20260927 15:07:24.699550 7239 external_mini_cluster.cc:1428] Setting key 3698be2607fc8e7ae4e2d97c2404f095
I20260927 15:07:24.705287 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.697906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:37679 (local address 127.7.17.254:40013)
0927 15:07:24.698013 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:24.698016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:24.698242 (+ 226us) server_negotiation.cc:407] Connection header received
0927 15:07:24.698406 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:24.698409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:24.698449 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:24.698516 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:24.699016 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.699744 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.700872 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.701105 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.701193 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.702271 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:24.702275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:24.702277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:24.702303 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:24.703752 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:24.704143 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:24.704146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:24.704148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:24.704227 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:24.704474 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:24.704477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:24.704479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:24.704651 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:24.704742 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:24.705023 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 15:07:24.705129 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":39}
I20260927 15:07:24.705291 9177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.697828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:40013 (local address 127.7.17.193:37679)
0927 15:07:24.698163 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:24.698178 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:24.698279 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:24.698551 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:24.698554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:24.698768 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:24.698894 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.698902 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.699874 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.699878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.700700 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.700710 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.701209 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.701212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.701304 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.701791 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:24.701803 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:24.702139 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:24.703853 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:24.703857 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:24.703860 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:24.704057 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:24.704303 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:24.704306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:24.704308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:24.704396 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:24.704740 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:24.704742 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:24.704860 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 15:07:24.704953 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 15:07:24.705099 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.7.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
I20260927 15:07:24.729210 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:40013
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:46797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:24.841956 9180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:24.842254 9180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:24.842303 9180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:24.846597 9180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:24.846674 9180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:24.846717 9180 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 15:07:24.846730 9180 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:07:24.846772 9180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:07:24.852319 9180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:46797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.17.254:40013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:24.853482 9180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:24.854540 9180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:07:24.864660 9180 server_base.cc:1034] running on GCE node
W20260927 15:07:24.864596 9185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:24.864603 9188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:24.864620 9186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:24.865303 9180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:24.865983 9180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:24.867154 9180 hybrid_clock.cc:648] HybridClock initialized: now 1790521644867128 us; error 38 us; skew 500 ppm
Sep 27 15:07:24 dist-test-slave-hhpq krb5kdc[8923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521644, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:24.869985 9180 init.cc:377] Logged in from keytab as kudu/127.7.17.194@KRBTEST.COM (short username kudu)
I20260927 15:07:24.870885 9180 webserver.cc:460] Webserver started at http://127.7.17.194:40707/ using document root <none> and password file <none>
I20260927 15:07:24.871501 9180 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:24.871568 9180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:24.871753 9180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:24.873860 9180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "e483d36712e14ba1be32dcacc0b65f4a"
format_stamp: "Formatted at 2026-09-27 15:07:24 on dist-test-slave-hhpq"
server_key: "5ab4d617f193627cbc06599135847e36"
server_key_iv: "0b61ab0f11e802e6df19e8b9356545cf"
server_key_version: "encryptionkey@0"
I20260927 15:07:24.874423 9180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "e483d36712e14ba1be32dcacc0b65f4a"
format_stamp: "Formatted at 2026-09-27 15:07:24 on dist-test-slave-hhpq"
server_key: "5ab4d617f193627cbc06599135847e36"
server_key_iv: "0b61ab0f11e802e6df19e8b9356545cf"
server_key_version: "encryptionkey@0"
I20260927 15:07:24.878192 9180 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:07:24.880751 9195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:24.882083 9180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:07:24.882200 9180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "e483d36712e14ba1be32dcacc0b65f4a"
format_stamp: "Formatted at 2026-09-27 15:07:24 on dist-test-slave-hhpq"
server_key: "5ab4d617f193627cbc06599135847e36"
server_key_iv: "0b61ab0f11e802e6df19e8b9356545cf"
server_key_version: "encryptionkey@0"
I20260927 15:07:24.882301 9180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:24.898756 9180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:24.902168 9180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:24.902371 9180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:24.903867 9180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:24.903921 9180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:24.903975 9180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:24.904004 9180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:24.914319 9180 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:32785
I20260927 15:07:24.914388 9310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:32785 every 8 connection(s)
Sep 27 15:07:24 dist-test-slave-hhpq krb5kdc[8923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521644, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:24.915378 9180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:07:24.915885 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 9180
I20260927 15:07:24.916003 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:07:24.916363 7239 external_mini_cluster.cc:1428] Setting key 709efc3ddbb94856962c73bb1fae541c
I20260927 15:07:24.919044 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.904799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:41703 (local address 127.7.17.254:40013)
0927 15:07:24.904959 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:24.904964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:24.905939 (+ 975us) server_negotiation.cc:407] Connection header received
0927 15:07:24.907309 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:24.907313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:24.907375 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:24.907455 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:24.909316 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.910020 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.911811 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.912057 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.912146 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.915236 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:24.915239 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:24.915241 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:24.915318 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:24.916949 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:24.917435 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:24.917438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:24.917440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:24.917524 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:24.917911 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:24.917914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:24.917915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:24.918097 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:24.918170 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:24.918823 (+ 653us) server_negotiation.cc:299] Negotiation successful
0927 15:07:24.918927 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 15:07:24.920054 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.905241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:40013 (local address 127.7.17.194:41703)
0927 15:07:24.905755 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:24.905817 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:24.907004 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:24.907668 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:24.907678 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:24.908258 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:24.909116 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.909142 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.910147 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.910150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.911672 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.911683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.912225 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.912229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.912459 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.913233 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:24.913253 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:24.914999 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:24.917054 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:24.917059 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:24.917071 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:24.917317 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:24.917619 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:24.917621 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:24.917623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:24.917812 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:24.918222 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:24.918229 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:24.918549 (+ 320us) client_negotiation.cc:769] Sending connection context
0927 15:07:24.918743 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 15:07:24.919110 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
I20260927 15:07:24.921413 9311 heartbeater.cc:347] Connected to a master server at 127.7.17.254:40013
I20260927 15:07:24.921579 9311 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:24.922188 9311 heartbeater.cc:511] Master 127.7.17.254:40013 requested a full tablet report, sending...
I20260927 15:07:24.923383 8967 ts_manager.cc:194] Registered new tserver with Master: e483d36712e14ba1be32dcacc0b65f4a (127.7.17.194:32785)
I20260927 15:07:24.923978 8967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:41703
WARNING: no policy specified for kudu/127.7.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.195@KRBTEST.COM" created.
I20260927 15:07:24.933358 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.925347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:40971 (local address 127.7.17.254:40013)
0927 15:07:24.925475 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:24.925479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:24.926493 (+ 1014us) server_negotiation.cc:407] Connection header received
0927 15:07:24.926617 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:24.926619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:24.926692 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:24.926765 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:24.927176 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.927826 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.928742 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.929074 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.929199 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.930410 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:24.930414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:24.930417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:24.930444 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:24.931753 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:24.932145 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:24.932149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:24.932151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:24.932224 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:24.932470 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:24.932474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:24.932475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:24.932694 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:24.932783 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:24.933076 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 15:07:24.933187 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:07:24.933367 9318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:24.925218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:40013 (local address 127.7.17.194:40971)
0927 15:07:24.926396 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:24.926411 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:24.926510 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:24.926786 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:24.926788 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:24.926975 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:24.927071 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.927077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.927940 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.927942 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.928627 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:24.928637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:24.929209 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:24.929212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:24.929298 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:24.929799 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:24.929853 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:24.930272 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:24.931861 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:24.931864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:24.931866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:24.932059 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:24.932305 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:24.932308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:24.932310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:24.932392 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:24.932786 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:24.932789 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:24.932921 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 15:07:24.933012 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 15:07:24.933166 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1087,"thread_start_us":94,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
I20260927 15:07:24.944530 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:40013
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:46797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:25.055153 9320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:25.055464 9320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:25.055514 9320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:25.059633 9320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:25.059718 9320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:25.059768 9320 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 15:07:25.059787 9320 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:07:25.059834 9320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:07:25.065271 9320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:46797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.17.254:40013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:25.066346 9320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:25.067348 9320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:25.077116 9328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:25.077155 9325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:25.077136 9326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:25.077183 9320 server_base.cc:1034] running on GCE node
I20260927 15:07:25.077894 9320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:25.078747 9320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:25.079923 9320 hybrid_clock.cc:648] HybridClock initialized: now 1790521645079902 us; error 34 us; skew 500 ppm
Sep 27 15:07:25 dist-test-slave-hhpq krb5kdc[8923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521645, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:25.082832 9320 init.cc:377] Logged in from keytab as kudu/127.7.17.195@KRBTEST.COM (short username kudu)
I20260927 15:07:25.083828 9320 webserver.cc:460] Webserver started at http://127.7.17.195:36009/ using document root <none> and password file <none>
I20260927 15:07:25.084398 9320 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:25.084460 9320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:25.084643 9320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:25.086838 9320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "01f11021a9954daea5ab11a7b5e5bbf4"
format_stamp: "Formatted at 2026-09-27 15:07:25 on dist-test-slave-hhpq"
server_key: "0266fcd4c38d4d3aceffbf04d93f882c"
server_key_iv: "c0dc6ec5fdcf218f80a68c042b1a2127"
server_key_version: "encryptionkey@0"
I20260927 15:07:25.087404 9320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "01f11021a9954daea5ab11a7b5e5bbf4"
format_stamp: "Formatted at 2026-09-27 15:07:25 on dist-test-slave-hhpq"
server_key: "0266fcd4c38d4d3aceffbf04d93f882c"
server_key_iv: "c0dc6ec5fdcf218f80a68c042b1a2127"
server_key_version: "encryptionkey@0"
I20260927 15:07:25.091137 9320 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:07:25.093556 9335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:25.094880 9320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:07:25.094976 9320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "01f11021a9954daea5ab11a7b5e5bbf4"
format_stamp: "Formatted at 2026-09-27 15:07:25 on dist-test-slave-hhpq"
server_key: "0266fcd4c38d4d3aceffbf04d93f882c"
server_key_iv: "c0dc6ec5fdcf218f80a68c042b1a2127"
server_key_version: "encryptionkey@0"
I20260927 15:07:25.095081 9320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:25.111620 9320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:25.114601 9320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:25.114777 9320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:25.116230 9320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:25.116281 9320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:25.116328 9320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:25.116356 9320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:07:25 dist-test-slave-hhpq krb5kdc[8923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521645, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:25.126344 9320 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:44007
I20260927 15:07:25.126703 9450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:44007 every 8 connection(s)
I20260927 15:07:25.127372 9320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:25.129734 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.117080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:59735 (local address 127.7.17.254:40013)
0927 15:07:25.117227 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:25.117232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:25.118244 (+ 1012us) server_negotiation.cc:407] Connection header received
0927 15:07:25.119544 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:25.119548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:25.119609 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:25.119705 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:25.121124 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.121831 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.123129 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.123349 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.123445 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.126055 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:25.126059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:25.126062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:25.126094 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:25.127543 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:25.128116 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:25.128120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:25.128121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:25.128220 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:25.128631 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:25.128634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:25.128636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:25.128897 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:25.128983 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:25.129455 (+ 472us) server_negotiation.cc:299] Negotiation successful
0927 15:07:25.129563 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:07:25.130483 9344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.117547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:40013 (local address 127.7.17.195:59735)
0927 15:07:25.118052 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:25.118114 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:25.119238 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:25.119919 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:25.119938 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:25.120356 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:25.120980 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.120998 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.121953 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.121957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.123017 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.123027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.123487 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.123491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.123665 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.124281 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:25.124296 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:25.125859 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:25.127653 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:25.127659 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:25.127675 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:25.128002 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:25.128318 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:25.128372 (+ 54us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:25.128374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:25.128542 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:25.128986 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:25.128992 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:25.129214 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 15:07:25.129415 (+ 201us) client_negotiation.cc:240] Negotiation successful
0927 15:07:25.129671 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":114,"threads_started":1}
I20260927 15:07:25.130891 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 9320
I20260927 15:07:25.131001 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:07:25.131306 7239 external_mini_cluster.cc:1428] Setting key 284cd6fee9a76710e4d5952ef315a206
I20260927 15:07:25.131875 9451 heartbeater.cc:347] Connected to a master server at 127.7.17.254:40013
I20260927 15:07:25.132023 9451 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:25.132634 9451 heartbeater.cc:511] Master 127.7.17.254:40013 requested a full tablet report, sending...
I20260927 15:07:25.133863 8968 ts_manager.cc:194] Registered new tserver with Master: 01f11021a9954daea5ab11a7b5e5bbf4 (127.7.17.195:44007)
I20260927 15:07:25.134523 8968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:59735
I20260927 15:07:25.140683 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.134074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:60663 (local address 127.7.17.254:40013)
0927 15:07:25.134187 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:25.134191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:25.134337 (+ 146us) server_negotiation.cc:407] Connection header received
0927 15:07:25.134438 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:25.134441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:25.134485 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:25.134552 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:25.134975 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.135699 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.136547 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.136755 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.136886 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.138027 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:25.138031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:25.138033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:25.138056 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:25.139268 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:25.139660 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:25.139664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:25.139665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:25.139743 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:25.139961 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:25.139964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:25.139965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:25.140127 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:25.140198 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:25.140476 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 15:07:25.140572 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 15:07:25.140686 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.134003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:40013 (local address 127.7.17.195:60663)
0927 15:07:25.134265 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:25.134277 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:25.134367 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:25.134566 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:25.134568 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:25.134771 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:25.134866 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.134872 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.135839 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.135843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.136435 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.136444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.136905 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.136909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.136992 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.137540 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:25.137551 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:25.137891 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:25.139377 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:25.139380 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:25.139381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:25.139573 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:25.139815 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:25.139818 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:25.139819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:25.139893 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:25.140232 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:25.140234 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:25.140320 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 15:07:25.140413 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 15:07:25.140532 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":101,"threads_started":1}
I20260927 15:07:25.145244 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:07:25 dist-test-slave-hhpq krb5kdc[8923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:07:25 dist-test-slave-hhpq krb5kdc[8923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:25.171134 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.161862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32930 (local address 127.7.17.254:40013)
0927 15:07:25.161979 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:25.161983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:25.161995 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:07:25.162035 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:25.162038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:25.162076 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:25.162133 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:25.162791 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.163300 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.164154 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.164404 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.164487 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.167017 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:25.167022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:25.167024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:25.167056 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:25.169146 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:25.169766 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:25.169788 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:25.169790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:25.169898 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:25.170288 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:25.170292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:25.170294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:25.170505 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:25.170624 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:25.170900 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 15:07:25.171004 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:07:25.174379 8968 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:07:25.177783 8968 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:25.193401 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.187563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33016 (local address 127.7.17.195:44007)
0927 15:07:25.188143 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:25.188155 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:25.188487 (+ 332us) server_negotiation.cc:407] Connection header received
0927 15:07:25.188553 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:25.188558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:25.188696 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:25.188928 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:25.189291 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.190477 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.191797 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.192396 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.192493 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.192649 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:25.193123 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 15:07:25.193244 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":92,"threads_started":1}
I20260927 15:07:25.193409 9468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.187176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:32785 (local address 127.0.0.1:39998)
0927 15:07:25.188020 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:25.188078 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:25.188217 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:25.188910 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:25.188916 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:25.188940 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:25.189079 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.189085 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.190596 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.190599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.191798 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.191805 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.192813 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.192816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.192963 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.192985 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:07:25.193051 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 15:07:25.193229 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":109,"thread_start_us":95,"threads_started":1}
I20260927 15:07:25.193409 9472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.187987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:33045 (local address 127.0.0.1:35472)
0927 15:07:25.188458 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:25.188469 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:25.188543 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:25.189253 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:25.189256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:25.189273 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:25.189368 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.189374 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.190261 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.190265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.191697 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.191707 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.192770 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.192773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.192882 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.192961 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 15:07:25.193077 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 15:07:25.193229 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":66,"thread_start_us":93,"threads_started":1}
I20260927 15:07:25.193634 9470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.187550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39998 (local address 127.7.17.194:32785)
0927 15:07:25.188341 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:25.188352 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:25.188377 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:07:25.188450 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:25.188456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:25.188697 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:25.189224 (+ 527us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:25.189385 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.190477 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.191911 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.192690 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.193178 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.193352 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:25.193403 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 15:07:25.193495 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":99,"threads_started":1}
I20260927 15:07:25.193862 9469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.187330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:44007 (local address 127.0.0.1:33016)
0927 15:07:25.188286 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:25.188298 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:25.188401 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:25.189068 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:25.189071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:25.189085 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:25.189166 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.189172 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.190633 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.190636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.191661 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.191670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.192700 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.192703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.192911 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.192961 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 15:07:25.193077 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 15:07:25.193756 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"mutex_wait_us":132,"thread_start_us":97,"threads_started":1}
I20260927 15:07:25.194446 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.188169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35472 (local address 127.7.17.193:33045)
0927 15:07:25.188768 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:25.188783 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:25.188805 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:07:25.188907 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:25.188913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:25.189112 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:25.189278 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:25.189452 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.190156 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.191842 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.192656 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.192768 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.192984 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:25.194220 (+ 1236us) server_negotiation.cc:299] Negotiation successful
0927 15:07:25.194317 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":83,"threads_started":1}
I20260927 15:07:25.196731 9105 tablet_service.cc:1467] Processing CreateTablet for tablet 747bd7918a0f45e69927c1002ca8f3b8 (DEFAULT_TABLE table=test-table [id=528945a8c5a8484eb4261ea543002c1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:25.196745 9245 tablet_service.cc:1467] Processing CreateTablet for tablet 747bd7918a0f45e69927c1002ca8f3b8 (DEFAULT_TABLE table=test-table [id=528945a8c5a8484eb4261ea543002c1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:25.196753 9385 tablet_service.cc:1467] Processing CreateTablet for tablet 747bd7918a0f45e69927c1002ca8f3b8 (DEFAULT_TABLE table=test-table [id=528945a8c5a8484eb4261ea543002c1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:25.197757 9105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 747bd7918a0f45e69927c1002ca8f3b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:25.197781 9245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 747bd7918a0f45e69927c1002ca8f3b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:25.197781 9385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 747bd7918a0f45e69927c1002ca8f3b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:25.203908 9474 tablet_bootstrap.cc:492] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5: Bootstrap starting.
I20260927 15:07:25.205484 9475 tablet_bootstrap.cc:492] T 747bd7918a0f45e69927c1002ca8f3b8 P e483d36712e14ba1be32dcacc0b65f4a: Bootstrap starting.
I20260927 15:07:25.207798 9475 tablet_bootstrap.cc:654] T 747bd7918a0f45e69927c1002ca8f3b8 P e483d36712e14ba1be32dcacc0b65f4a: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:25.207787 9474 tablet_bootstrap.cc:654] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:25.208729 9474 log.cc:824] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:25.208729 9475 log.cc:824] T 747bd7918a0f45e69927c1002ca8f3b8 P e483d36712e14ba1be32dcacc0b65f4a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:25.209295 9476 tablet_bootstrap.cc:492] T 747bd7918a0f45e69927c1002ca8f3b8 P 01f11021a9954daea5ab11a7b5e5bbf4: Bootstrap starting.
I20260927 15:07:25.210945 9474 tablet_bootstrap.cc:492] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5: No bootstrap required, opened a new log
I20260927 15:07:25.210945 9475 tablet_bootstrap.cc:492] T 747bd7918a0f45e69927c1002ca8f3b8 P e483d36712e14ba1be32dcacc0b65f4a: No bootstrap required, opened a new log
I20260927 15:07:25.211160 9475 ts_tablet_manager.cc:1397] T 747bd7918a0f45e69927c1002ca8f3b8 P e483d36712e14ba1be32dcacc0b65f4a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:07:25.211160 9474 ts_tablet_manager.cc:1397] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260927 15:07:25.211498 9476 tablet_bootstrap.cc:654] T 747bd7918a0f45e69927c1002ca8f3b8 P 01f11021a9954daea5ab11a7b5e5bbf4: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:25.212203 9476 log.cc:824] T 747bd7918a0f45e69927c1002ca8f3b8 P 01f11021a9954daea5ab11a7b5e5bbf4: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:25.214003 9476 tablet_bootstrap.cc:492] T 747bd7918a0f45e69927c1002ca8f3b8 P 01f11021a9954daea5ab11a7b5e5bbf4: No bootstrap required, opened a new log
I20260927 15:07:25.214210 9476 ts_tablet_manager.cc:1397] T 747bd7918a0f45e69927c1002ca8f3b8 P 01f11021a9954daea5ab11a7b5e5bbf4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:07:25.215188 9474 raft_consensus.cc:348] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } } peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } }
I20260927 15:07:25.215189 9475 raft_consensus.cc:348] T 747bd7918a0f45e69927c1002ca8f3b8 P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } } peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } }
I20260927 15:07:25.215483 9474 raft_consensus.cc:374] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:25.215483 9475 raft_consensus.cc:374] T 747bd7918a0f45e69927c1002ca8f3b8 P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:25.215539 9474 raft_consensus.cc:729] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e676da6006f24776bc62de5cd993cee5, State: Initialized, Role: FOLLOWER
I20260927 15:07:25.215546 9475 raft_consensus.cc:729] T 747bd7918a0f45e69927c1002ca8f3b8 P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e483d36712e14ba1be32dcacc0b65f4a, State: Initialized, Role: FOLLOWER
I20260927 15:07:25.216017 9475 consensus_queue.cc:260] T 747bd7918a0f45e69927c1002ca8f3b8 P e483d36712e14ba1be32dcacc0b65f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } } peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } }
I20260927 15:07:25.216017 9474 consensus_queue.cc:260] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } } peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } }
I20260927 15:07:25.216732 9311 heartbeater.cc:503] Master 127.7.17.254:40013 was elected leader, sending a full tablet report...
I20260927 15:07:25.216998 9171 heartbeater.cc:503] Master 127.7.17.254:40013 was elected leader, sending a full tablet report...
I20260927 15:07:25.217111 9475 ts_tablet_manager.cc:1428] T 747bd7918a0f45e69927c1002ca8f3b8 P e483d36712e14ba1be32dcacc0b65f4a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 15:07:25.217985 9474 ts_tablet_manager.cc:1428] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260927 15:07:25.218918 9476 raft_consensus.cc:348] T 747bd7918a0f45e69927c1002ca8f3b8 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } } peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } }
I20260927 15:07:25.219161 9476 raft_consensus.cc:374] T 747bd7918a0f45e69927c1002ca8f3b8 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:25.219205 9476 raft_consensus.cc:729] T 747bd7918a0f45e69927c1002ca8f3b8 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01f11021a9954daea5ab11a7b5e5bbf4, State: Initialized, Role: FOLLOWER
I20260927 15:07:25.219663 9476 consensus_queue.cc:260] T 747bd7918a0f45e69927c1002ca8f3b8 P 01f11021a9954daea5ab11a7b5e5bbf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } } peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } }
I20260927 15:07:25.220495 9451 heartbeater.cc:503] Master 127.7.17.254:40013 was elected leader, sending a full tablet report...
I20260927 15:07:25.221304 9476 ts_tablet_manager.cc:1428] T 747bd7918a0f45e69927c1002ca8f3b8 P 01f11021a9954daea5ab11a7b5e5bbf4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260927 15:07:25.378317 9452 tablet.cc:2369] T 747bd7918a0f45e69927c1002ca8f3b8 P 01f11021a9954daea5ab11a7b5e5bbf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:07:25.417047 9312 tablet.cc:2369] T 747bd7918a0f45e69927c1002ca8f3b8 P e483d36712e14ba1be32dcacc0b65f4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:07:25.442821 9172 tablet.cc:2369] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:07:25.494608 9481 raft_consensus.cc:482] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:25.494830 9481 raft_consensus.cc:504] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } } peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } }
I20260927 15:07:25.496038 9481 leader_election.cc:288] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e483d36712e14ba1be32dcacc0b65f4a (127.7.17.194:32785), 01f11021a9954daea5ab11a7b5e5bbf4 (127.7.17.195:44007)
I20260927 15:07:25.501261 9470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.496359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:46191 (local address 127.7.17.194:32785)
0927 15:07:25.496613 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:25.496617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:25.496654 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:07:25.496820 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:25.496847 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:25.496902 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:25.496989 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:25.497486 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.498209 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.499963 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.500608 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.500704 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.500750 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:25.501046 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 15:07:25.501128 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260927 15:07:25.501261 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.496431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:56623 (local address 127.7.17.195:44007)
0927 15:07:25.496556 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:25.496560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:25.496749 (+ 189us) server_negotiation.cc:407] Connection header received
0927 15:07:25.496858 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:25.496860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:25.496914 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:25.496998 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:25.497358 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.498210 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.499963 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.500565 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.500697 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.500750 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:25.501046 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 15:07:25.501128 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:07:25.501334 9483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.496266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:32785 (local address 127.7.17.193:46191)
0927 15:07:25.496572 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:25.496588 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:25.496714 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:25.497020 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:25.497022 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:25.497065 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:25.497253 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.497260 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.498392 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.498395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.499831 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.499837 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.500703 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.500706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.500854 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.500912 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 15:07:25.500986 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 15:07:25.501121 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":255,"spinlock_wait_cycles":6016,"thread_start_us":116,"threads_started":1}
I20260927 15:07:25.501338 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.496294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:44007 (local address 127.7.17.193:56623)
0927 15:07:25.496674 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:25.496684 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:25.496761 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:25.497019 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:25.497022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:25.497064 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:25.497252 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.497260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.498347 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.498351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.499821 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.499835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.500697 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.500701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.500873 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.500907 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 15:07:25.500986 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 15:07:25.501122 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":103,"spinlock_wait_cycles":28416,"thread_start_us":99,"threads_started":1}
I20260927 15:07:25.502103 9405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747bd7918a0f45e69927c1002ca8f3b8" candidate_uuid: "e676da6006f24776bc62de5cd993cee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" is_pre_election: true
I20260927 15:07:25.502177 9265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747bd7918a0f45e69927c1002ca8f3b8" candidate_uuid: "e676da6006f24776bc62de5cd993cee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e483d36712e14ba1be32dcacc0b65f4a" is_pre_election: true
I20260927 15:07:25.502415 9405 raft_consensus.cc:2454] T 747bd7918a0f45e69927c1002ca8f3b8 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e676da6006f24776bc62de5cd993cee5 in term 0.
I20260927 15:07:25.502444 9265 raft_consensus.cc:2454] T 747bd7918a0f45e69927c1002ca8f3b8 P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e676da6006f24776bc62de5cd993cee5 in term 0.
I20260927 15:07:25.502895 9056 leader_election.cc:302] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f11021a9954daea5ab11a7b5e5bbf4, e676da6006f24776bc62de5cd993cee5; no voters:
I20260927 15:07:25.503155 9481 raft_consensus.cc:2781] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:25.503243 9481 raft_consensus.cc:482] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:25.503358 9481 raft_consensus.cc:3037] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:25.504350 9481 raft_consensus.cc:504] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } } peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } }
I20260927 15:07:25.504679 9481 leader_election.cc:288] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5 [CANDIDATE]: Term 1 election: Requested vote from peers e483d36712e14ba1be32dcacc0b65f4a (127.7.17.194:32785), 01f11021a9954daea5ab11a7b5e5bbf4 (127.7.17.195:44007)
I20260927 15:07:25.505091 9265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747bd7918a0f45e69927c1002ca8f3b8" candidate_uuid: "e676da6006f24776bc62de5cd993cee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e483d36712e14ba1be32dcacc0b65f4a"
I20260927 15:07:25.505236 9265 raft_consensus.cc:3037] T 747bd7918a0f45e69927c1002ca8f3b8 P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:25.505259 9405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747bd7918a0f45e69927c1002ca8f3b8" candidate_uuid: "e676da6006f24776bc62de5cd993cee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01f11021a9954daea5ab11a7b5e5bbf4"
I20260927 15:07:25.505405 9405 raft_consensus.cc:3037] T 747bd7918a0f45e69927c1002ca8f3b8 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:25.506330 9265 raft_consensus.cc:2454] T 747bd7918a0f45e69927c1002ca8f3b8 P e483d36712e14ba1be32dcacc0b65f4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e676da6006f24776bc62de5cd993cee5 in term 1.
I20260927 15:07:25.506649 9059 leader_election.cc:302] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e483d36712e14ba1be32dcacc0b65f4a, e676da6006f24776bc62de5cd993cee5; no voters:
I20260927 15:07:25.506747 9405 raft_consensus.cc:2454] T 747bd7918a0f45e69927c1002ca8f3b8 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e676da6006f24776bc62de5cd993cee5 in term 1.
I20260927 15:07:25.506817 9481 raft_consensus.cc:2781] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:25.507057 9481 raft_consensus.cc:686] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5 [term 1 LEADER]: Becoming Leader. State: Replica: e676da6006f24776bc62de5cd993cee5, State: Running, Role: LEADER
I20260927 15:07:25.507463 9481 consensus_queue.cc:237] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } } peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } }
I20260927 15:07:25.510620 8967 catalog_manager.cc:5495] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e676da6006f24776bc62de5cd993cee5 (127.7.17.193). New cstate: current_term: 1 leader_uuid: "e676da6006f24776bc62de5cd993cee5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } health_report { overall_health: HEALTHY } } }
Sep 27 15:07:25 dist-test-slave-hhpq krb5kdc[8923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521644, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:25.602873 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.594698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32934 (local address 127.7.17.254:40013)
0927 15:07:25.594813 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:25.594818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:25.595049 (+ 231us) server_negotiation.cc:407] Connection header received
0927 15:07:25.595175 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:25.595178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:25.595213 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:25.595276 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:25.596032 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.596549 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.597359 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.597557 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.597637 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.599982 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:25.599985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:25.599987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:25.600008 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:25.601230 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:25.601662 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:25.601665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:25.601667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:25.601750 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:25.601983 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:25.601986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:25.601988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:25.602158 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:25.602240 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:25.602571 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 15:07:25.602721 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":37}
I20260927 15:07:25.602885 9492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.594638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:40013 (local address 127.0.0.1:32934)
0927 15:07:25.594954 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:25.594968 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:25.595088 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:25.595308 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:25.595310 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:25.595831 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:25.595927 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.595934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.596646 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.596649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.597257 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.597267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.597648 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.597651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.597779 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.598315 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:25.598326 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:25.599837 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:25.601330 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:25.601340 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:25.601349 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:25.601578 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:25.601826 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:25.601829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:25.601831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:25.601908 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:25.602292 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:25.602300 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:25.602417 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 15:07:25.602512 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 15:07:25.602674 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":31,"thread_start_us":121,"threads_started":1}
I20260927 15:07:25.606844 8967 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:32934:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 15:07:25.607671 8967 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:25.612511 9385 tablet_service.cc:1467] Processing CreateTablet for tablet 45b274ef45aa4c2398ef129b7bd3fdcf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ed46961bf5334bc0ba13c13e22fefc98]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:07:25.612519 9245 tablet_service.cc:1467] Processing CreateTablet for tablet 45b274ef45aa4c2398ef129b7bd3fdcf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ed46961bf5334bc0ba13c13e22fefc98]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:07:25.612519 9105 tablet_service.cc:1467] Processing CreateTablet for tablet 45b274ef45aa4c2398ef129b7bd3fdcf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ed46961bf5334bc0ba13c13e22fefc98]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:07:25.613255 9385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45b274ef45aa4c2398ef129b7bd3fdcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:25.613253 9105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45b274ef45aa4c2398ef129b7bd3fdcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:25.613253 9245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45b274ef45aa4c2398ef129b7bd3fdcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:25.615664 9476 tablet_bootstrap.cc:492] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4: Bootstrap starting.
I20260927 15:07:25.616094 9474 tablet_bootstrap.cc:492] T 45b274ef45aa4c2398ef129b7bd3fdcf P e676da6006f24776bc62de5cd993cee5: Bootstrap starting.
I20260927 15:07:25.616693 9476 tablet_bootstrap.cc:654] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:25.617211 9474 tablet_bootstrap.cc:654] T 45b274ef45aa4c2398ef129b7bd3fdcf P e676da6006f24776bc62de5cd993cee5: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:25.617365 9475 tablet_bootstrap.cc:492] T 45b274ef45aa4c2398ef129b7bd3fdcf P e483d36712e14ba1be32dcacc0b65f4a: Bootstrap starting.
I20260927 15:07:25.617937 9476 tablet_bootstrap.cc:492] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4: No bootstrap required, opened a new log
I20260927 15:07:25.618023 9476 ts_tablet_manager.cc:1397] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:07:25.618225 9475 tablet_bootstrap.cc:654] T 45b274ef45aa4c2398ef129b7bd3fdcf P e483d36712e14ba1be32dcacc0b65f4a: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:25.618382 9474 tablet_bootstrap.cc:492] T 45b274ef45aa4c2398ef129b7bd3fdcf P e676da6006f24776bc62de5cd993cee5: No bootstrap required, opened a new log
I20260927 15:07:25.618467 9474 ts_tablet_manager.cc:1397] T 45b274ef45aa4c2398ef129b7bd3fdcf P e676da6006f24776bc62de5cd993cee5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:07:25.618461 9476 raft_consensus.cc:348] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:25.618587 9476 raft_consensus.cc:374] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:25.618621 9476 raft_consensus.cc:729] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01f11021a9954daea5ab11a7b5e5bbf4, State: Initialized, Role: FOLLOWER
I20260927 15:07:25.618777 9476 consensus_queue.cc:260] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:25.618935 9474 raft_consensus.cc:348] T 45b274ef45aa4c2398ef129b7bd3fdcf P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:25.619035 9476 ts_tablet_manager.cc:1428] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:25.619045 9474 raft_consensus.cc:374] T 45b274ef45aa4c2398ef129b7bd3fdcf P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:25.619076 9474 raft_consensus.cc:729] T 45b274ef45aa4c2398ef129b7bd3fdcf P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e676da6006f24776bc62de5cd993cee5, State: Initialized, Role: FOLLOWER
I20260927 15:07:25.619181 9474 consensus_queue.cc:260] T 45b274ef45aa4c2398ef129b7bd3fdcf P e676da6006f24776bc62de5cd993cee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:25.619426 9474 ts_tablet_manager.cc:1428] T 45b274ef45aa4c2398ef129b7bd3fdcf P e676da6006f24776bc62de5cd993cee5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:25.619683 9475 tablet_bootstrap.cc:492] T 45b274ef45aa4c2398ef129b7bd3fdcf P e483d36712e14ba1be32dcacc0b65f4a: No bootstrap required, opened a new log
I20260927 15:07:25.619661 9482 tablet_replica.cc:439] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } } }
I20260927 15:07:25.619762 9475 ts_tablet_manager.cc:1397] T 45b274ef45aa4c2398ef129b7bd3fdcf P e483d36712e14ba1be32dcacc0b65f4a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:07:25.619812 9482 tablet_replica.cc:442] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:25.620033 9481 tablet_replica.cc:439] T 45b274ef45aa4c2398ef129b7bd3fdcf P e676da6006f24776bc62de5cd993cee5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } } }
I20260927 15:07:25.620151 9481 tablet_replica.cc:442] T 45b274ef45aa4c2398ef129b7bd3fdcf P e676da6006f24776bc62de5cd993cee5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:25.620179 9475 raft_consensus.cc:348] T 45b274ef45aa4c2398ef129b7bd3fdcf P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:25.620276 9475 raft_consensus.cc:374] T 45b274ef45aa4c2398ef129b7bd3fdcf P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:25.620311 9475 raft_consensus.cc:729] T 45b274ef45aa4c2398ef129b7bd3fdcf P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e483d36712e14ba1be32dcacc0b65f4a, State: Initialized, Role: FOLLOWER
I20260927 15:07:25.620448 9475 consensus_queue.cc:260] T 45b274ef45aa4c2398ef129b7bd3fdcf P e483d36712e14ba1be32dcacc0b65f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:25.620689 9475 ts_tablet_manager.cc:1428] T 45b274ef45aa4c2398ef129b7bd3fdcf P e483d36712e14ba1be32dcacc0b65f4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 15:07:25.620700 9480 tablet_replica.cc:439] T 45b274ef45aa4c2398ef129b7bd3fdcf P e483d36712e14ba1be32dcacc0b65f4a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } } }
I20260927 15:07:25.621011 9480 tablet_replica.cc:442] T 45b274ef45aa4c2398ef129b7bd3fdcf P e483d36712e14ba1be32dcacc0b65f4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:25.647135 9482 raft_consensus.cc:482] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:25.647262 9482 raft_consensus.cc:504] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:25.648331 9482 leader_election.cc:288] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e676da6006f24776bc62de5cd993cee5 (127.7.17.193:33045), e483d36712e14ba1be32dcacc0b65f4a (127.7.17.194:32785)
I20260927 15:07:25.652274 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.648679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:60679 (local address 127.7.17.193:33045)
0927 15:07:25.648840 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:25.648845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:25.649250 (+ 405us) server_negotiation.cc:407] Connection header received
0927 15:07:25.649435 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:25.649438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:25.649490 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:25.649550 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:25.649792 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.650306 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.651351 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.651747 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.651850 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.651898 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:25.652074 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 15:07:25.652141 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:07:25.652292 9496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.648575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:32785 (local address 127.7.17.195:50327)
0927 15:07:25.648867 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:25.648882 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:25.648993 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:25.649249 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:25.649252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:25.649271 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:25.649398 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.649405 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.650228 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.650232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.651185 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.651195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.651841 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.651844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.651960 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.651983 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:07:25.652047 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 15:07:25.652142 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":71,"spinlock_wait_cycles":30592,"thread_start_us":67,"threads_started":1}
I20260927 15:07:25.652274 9470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.648702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:50327 (local address 127.7.17.194:32785)
0927 15:07:25.648841 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:25.648845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:25.648987 (+ 142us) server_negotiation.cc:407] Connection header received
0927 15:07:25.649085 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:25.649089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:25.649140 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:25.649229 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:25.649502 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.650127 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.651322 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.651725 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.651833 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.651886 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:25.652077 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 15:07:25.652147 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:07:25.652536 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.648577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:33045 (local address 127.7.17.195:60679)
0927 15:07:25.649142 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:25.649155 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:25.649331 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:25.649571 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:25.649574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:25.649592 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:25.649693 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.649699 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.650463 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.650466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.651218 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.651226 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.651849 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.651853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.651969 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.651986 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 15:07:25.652048 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 15:07:25.652147 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":223,"thread_start_us":121,"threads_started":1}
I20260927 15:07:25.652731 9265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b274ef45aa4c2398ef129b7bd3fdcf" candidate_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e483d36712e14ba1be32dcacc0b65f4a" is_pre_election: true
I20260927 15:07:25.652922 9265 raft_consensus.cc:2454] T 45b274ef45aa4c2398ef129b7bd3fdcf P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f11021a9954daea5ab11a7b5e5bbf4 in term 0.
I20260927 15:07:25.653141 9125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b274ef45aa4c2398ef129b7bd3fdcf" candidate_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e676da6006f24776bc62de5cd993cee5" is_pre_election: true
I20260927 15:07:25.653280 9339 leader_election.cc:302] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f11021a9954daea5ab11a7b5e5bbf4, e483d36712e14ba1be32dcacc0b65f4a; no voters:
I20260927 15:07:25.653383 9125 raft_consensus.cc:2454] T 45b274ef45aa4c2398ef129b7bd3fdcf P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f11021a9954daea5ab11a7b5e5bbf4 in term 0.
I20260927 15:07:25.653525 9482 raft_consensus.cc:2781] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:25.653585 9482 raft_consensus.cc:482] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:25.653630 9482 raft_consensus.cc:3037] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:25.654840 9482 raft_consensus.cc:504] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:25.655225 9482 leader_election.cc:288] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4 [CANDIDATE]: Term 1 election: Requested vote from peers e676da6006f24776bc62de5cd993cee5 (127.7.17.193:33045), e483d36712e14ba1be32dcacc0b65f4a (127.7.17.194:32785)
I20260927 15:07:25.655490 9125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b274ef45aa4c2398ef129b7bd3fdcf" candidate_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e676da6006f24776bc62de5cd993cee5"
I20260927 15:07:25.655607 9125 raft_consensus.cc:3037] T 45b274ef45aa4c2398ef129b7bd3fdcf P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:25.655606 9265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b274ef45aa4c2398ef129b7bd3fdcf" candidate_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e483d36712e14ba1be32dcacc0b65f4a"
I20260927 15:07:25.655740 9265 raft_consensus.cc:3037] T 45b274ef45aa4c2398ef129b7bd3fdcf P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:25.656744 9265 raft_consensus.cc:2454] T 45b274ef45aa4c2398ef129b7bd3fdcf P e483d36712e14ba1be32dcacc0b65f4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f11021a9954daea5ab11a7b5e5bbf4 in term 1.
I20260927 15:07:25.656744 9125 raft_consensus.cc:2454] T 45b274ef45aa4c2398ef129b7bd3fdcf P e676da6006f24776bc62de5cd993cee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f11021a9954daea5ab11a7b5e5bbf4 in term 1.
I20260927 15:07:25.657039 9336 leader_election.cc:302] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f11021a9954daea5ab11a7b5e5bbf4, e676da6006f24776bc62de5cd993cee5; no voters:
I20260927 15:07:25.657194 9482 raft_consensus.cc:2781] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:25.657397 9482 raft_consensus.cc:686] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4 [term 1 LEADER]: Becoming Leader. State: Replica: 01f11021a9954daea5ab11a7b5e5bbf4, State: Running, Role: LEADER
I20260927 15:07:25.657732 9482 consensus_queue.cc:237] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:25.659941 9498 tablet_replica.cc:439] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4: TxnStatusTablet state changed. Reason: New leader 01f11021a9954daea5ab11a7b5e5bbf4. Latest consensus state: current_term: 1 leader_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } } }
I20260927 15:07:25.660074 9498 tablet_replica.cc:442] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:07:25.660363 9500 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 15:07:25.660586 9500 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 15:07:25.661324 8968 catalog_manager.cc:5495] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01f11021a9954daea5ab11a7b5e5bbf4 (127.7.17.195). New cstate: current_term: 1 leader_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } health_report { overall_health: HEALTHY } } }
I20260927 15:07:25.677392 9492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.672750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:44007 (local address 127.0.0.1:33018)
0927 15:07:25.673086 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:25.673099 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:25.673204 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:25.673569 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:25.673571 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:25.673579 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:07:25.673696 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.673701 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.674800 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.674803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.675453 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.675462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.676036 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.676039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.676133 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.676188 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:07:25.677084 (+ 896us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:07:25.677104 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 15:07:25.677163 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 15:07:25.677251 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237}
I20260927 15:07:25.677400 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.672936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33018 (local address 127.7.17.195:44007)
0927 15:07:25.673084 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:25.673089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:25.673145 (+ 56us) server_negotiation.cc:407] Connection header received
0927 15:07:25.673342 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:25.673345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:25.673409 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:25.673517 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:25.673802 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.674663 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.675568 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.675909 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.676010 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.676341 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:25.676964 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:25.677081 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:25.677187 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 15:07:25.677252 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 15:07:25.683539 9265 raft_consensus.cc:1261] T 45b274ef45aa4c2398ef129b7bd3fdcf P e483d36712e14ba1be32dcacc0b65f4a [term 1 FOLLOWER]: Refusing update from remote peer 01f11021a9954daea5ab11a7b5e5bbf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:25.683538 9125 raft_consensus.cc:1261] T 45b274ef45aa4c2398ef129b7bd3fdcf P e676da6006f24776bc62de5cd993cee5 [term 1 FOLLOWER]: Refusing update from remote peer 01f11021a9954daea5ab11a7b5e5bbf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:25.684288 9482 consensus_queue.cc:1035] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:25.684484 9498 consensus_queue.cc:1035] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:25.688097 9481 tablet_replica.cc:439] T 45b274ef45aa4c2398ef129b7bd3fdcf P e676da6006f24776bc62de5cd993cee5: TxnStatusTablet state changed. Reason: New leader 01f11021a9954daea5ab11a7b5e5bbf4. Latest consensus state: current_term: 1 leader_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } } }
I20260927 15:07:25.688257 9481 tablet_replica.cc:442] T 45b274ef45aa4c2398ef129b7bd3fdcf P e676da6006f24776bc62de5cd993cee5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:25.688313 9480 tablet_replica.cc:439] T 45b274ef45aa4c2398ef129b7bd3fdcf P e483d36712e14ba1be32dcacc0b65f4a: TxnStatusTablet state changed. Reason: New leader 01f11021a9954daea5ab11a7b5e5bbf4. Latest consensus state: current_term: 1 leader_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } } }
I20260927 15:07:25.688442 9480 tablet_replica.cc:442] T 45b274ef45aa4c2398ef129b7bd3fdcf P e483d36712e14ba1be32dcacc0b65f4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:25.691862 9498 tablet_replica.cc:439] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } } }
I20260927 15:07:25.692001 9498 tablet_replica.cc:442] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:07:25.692296 9481 tablet_replica.cc:439] T 45b274ef45aa4c2398ef129b7bd3fdcf P e676da6006f24776bc62de5cd993cee5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } } }
I20260927 15:07:25.692417 9481 tablet_replica.cc:442] T 45b274ef45aa4c2398ef129b7bd3fdcf P e676da6006f24776bc62de5cd993cee5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:25.696986 9482 tablet_replica.cc:439] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } } }
I20260927 15:07:25.697122 9482 tablet_replica.cc:442] T 45b274ef45aa4c2398ef129b7bd3fdcf P 01f11021a9954daea5ab11a7b5e5bbf4: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:07:25.699914 9505 mvcc.cc:205] Tried to move back new op lower bound from 7333976660712304640 to 7333976660616937472. Current Snapshot: MvccSnapshot[applied={T|T < 7333976660712304640}]
I20260927 15:07:25.700407 9480 tablet_replica.cc:439] T 45b274ef45aa4c2398ef129b7bd3fdcf P e483d36712e14ba1be32dcacc0b65f4a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } } }
I20260927 15:07:25.700532 9480 tablet_replica.cc:442] T 45b274ef45aa4c2398ef129b7bd3fdcf P e483d36712e14ba1be32dcacc0b65f4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:25.707020 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.698252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.7.17.193:33045)
0927 15:07:25.698376 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:25.698380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:25.702696 (+ 4316us) server_negotiation.cc:407] Connection header received
0927 15:07:25.702824 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:25.702828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:25.702892 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:25.703015 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:25.703277 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.704068 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.705352 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.705700 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.705802 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.706069 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:25.706629 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:25.706736 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:25.706821 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 15:07:25.706880 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:07:25.718312 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.715110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:34843 (local address 127.7.17.195:44007)
0927 15:07:25.715250 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:25.715255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:25.715421 (+ 166us) server_negotiation.cc:407] Connection header received
0927 15:07:25.715561 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:25.715564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:25.715612 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:25.715672 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:25.715903 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.716399 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.717286 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.717496 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.717579 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.717894 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:25.717964 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:25.718018 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:25.718117 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 15:07:25.718167 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:07:25.718330 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.714988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:44007 (local address 127.7.17.193:34843)
0927 15:07:25.715334 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:25.715347 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:25.715461 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:25.715688 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:25.715691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:25.715698 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:07:25.715801 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.715807 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.716495 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.716498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.717179 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.717188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.717588 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.717592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.717671 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.717713 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:07:25.718040 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:07:25.718047 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:07:25.718100 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 15:07:25.718179 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":106,"threads_started":1}
I20260927 15:07:25.726491 9265 raft_consensus.cc:1261] T 747bd7918a0f45e69927c1002ca8f3b8 P e483d36712e14ba1be32dcacc0b65f4a [term 1 FOLLOWER]: Refusing update from remote peer e676da6006f24776bc62de5cd993cee5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:25.726650 9405 raft_consensus.cc:1261] T 747bd7918a0f45e69927c1002ca8f3b8 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 1 FOLLOWER]: Refusing update from remote peer e676da6006f24776bc62de5cd993cee5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:25.727110 9485 consensus_queue.cc:1035] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:25.727332 9481 consensus_queue.cc:1035] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:25.751176 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.747693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:43639 (local address 127.7.17.193:33045)
0927 15:07:25.747806 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:25.747811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:25.748093 (+ 282us) server_negotiation.cc:407] Connection header received
0927 15:07:25.748211 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:25.748214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:25.748264 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:25.748324 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:25.748564 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.749181 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.750104 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.750327 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.750420 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.750732 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:25.750809 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:25.750867 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:25.750998 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 15:07:25.751058 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:07:25.751583 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.747616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:33045 (local address 127.7.17.195:43639)
0927 15:07:25.748007 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:25.748020 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:25.748131 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:25.748341 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:25.748344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:25.748351 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:07:25.748472 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.748478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.749319 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.749323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.749990 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.750000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.750428 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.750431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.750525 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.750582 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:07:25.750892 (+ 310us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:07:25.750899 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:07:25.751374 (+ 475us) client_negotiation.cc:240] Negotiation successful
0927 15:07:25.751458 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":109,"threads_started":1}
I20260927 15:07:25.775827 8967 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:25.776044 8967 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:25.778995 8967 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1: Sending DeleteTablet for 3 replicas of tablet 747bd7918a0f45e69927c1002ca8f3b8
I20260927 15:07:25.779837 9245 tablet_service.cc:1514] Processing DeleteTablet for tablet 747bd7918a0f45e69927c1002ca8f3b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:25 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:39998
I20260927 15:07:25.780118 9105 tablet_service.cc:1514] Processing DeleteTablet for tablet 747bd7918a0f45e69927c1002ca8f3b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:25 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:35472
I20260927 15:07:25.780628 9525 tablet_replica.cc:330] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5: stopping tablet replica
I20260927 15:07:25.780776 9385 tablet_service.cc:1514] Processing DeleteTablet for tablet 747bd7918a0f45e69927c1002ca8f3b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:25 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:33016
I20260927 15:07:25.781145 9525 raft_consensus.cc:2229] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:07:25.781603 9525 raft_consensus.cc:2258] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:25.782123 8967 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:07:25.782675 8967 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:25.783385 9525 ts_tablet_manager.cc:1905] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:25.784291 9526 tablet_replica.cc:330] T 747bd7918a0f45e69927c1002ca8f3b8 P 01f11021a9954daea5ab11a7b5e5bbf4: stopping tablet replica
I20260927 15:07:25.784623 9526 raft_consensus.cc:2229] T 747bd7918a0f45e69927c1002ca8f3b8 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:25.784785 9526 raft_consensus.cc:2258] T 747bd7918a0f45e69927c1002ca8f3b8 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:25.789028 9524 tablet_replica.cc:330] T 747bd7918a0f45e69927c1002ca8f3b8 P e483d36712e14ba1be32dcacc0b65f4a: stopping tablet replica
I20260927 15:07:25.789088 9525 ts_tablet_manager.cc:1918] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:07:25.789156 9245 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae27503cf9c475aa34700f456ef87c1 (DEFAULT_TABLE table=test-table [id=5b0b5f9e74f54be488bb17164e13f3f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:25.789356 9525 log.cc:1196] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-0/wal/wals/747bd7918a0f45e69927c1002ca8f3b8
I20260927 15:07:25.789404 9524 raft_consensus.cc:2229] T 747bd7918a0f45e69927c1002ca8f3b8 P e483d36712e14ba1be32dcacc0b65f4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:25.789453 9245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae27503cf9c475aa34700f456ef87c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:25.789829 9525 ts_tablet_manager.cc:1939] T 747bd7918a0f45e69927c1002ca8f3b8 P e676da6006f24776bc62de5cd993cee5: Deleting consensus metadata
I20260927 15:07:25.790381 9105 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae27503cf9c475aa34700f456ef87c1 (DEFAULT_TABLE table=test-table [id=5b0b5f9e74f54be488bb17164e13f3f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:25.790637 9105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae27503cf9c475aa34700f456ef87c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:25.791239 8952 catalog_manager.cc:4841] TS e676da6006f24776bc62de5cd993cee5 (127.7.17.193:33045): tablet 747bd7918a0f45e69927c1002ca8f3b8 (table test-table [id=528945a8c5a8484eb4261ea543002c1d]) successfully deleted
I20260927 15:07:25.792223 9475 tablet_bootstrap.cc:492] T 5ae27503cf9c475aa34700f456ef87c1 P e483d36712e14ba1be32dcacc0b65f4a: Bootstrap starting.
I20260927 15:07:25.792464 9524 raft_consensus.cc:2258] T 747bd7918a0f45e69927c1002ca8f3b8 P e483d36712e14ba1be32dcacc0b65f4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:25.793131 9385 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae27503cf9c475aa34700f456ef87c1 (DEFAULT_TABLE table=test-table [id=5b0b5f9e74f54be488bb17164e13f3f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:25.793432 9385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae27503cf9c475aa34700f456ef87c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:25.793498 9475 tablet_bootstrap.cc:654] T 5ae27503cf9c475aa34700f456ef87c1 P e483d36712e14ba1be32dcacc0b65f4a: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:25.794884 9475 tablet_bootstrap.cc:492] T 5ae27503cf9c475aa34700f456ef87c1 P e483d36712e14ba1be32dcacc0b65f4a: No bootstrap required, opened a new log
I20260927 15:07:25.794972 9475 ts_tablet_manager.cc:1397] T 5ae27503cf9c475aa34700f456ef87c1 P e483d36712e14ba1be32dcacc0b65f4a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 15:07:25.795428 9475 raft_consensus.cc:348] T 5ae27503cf9c475aa34700f456ef87c1 P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:25.795539 9475 raft_consensus.cc:374] T 5ae27503cf9c475aa34700f456ef87c1 P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:25.795568 9475 raft_consensus.cc:729] T 5ae27503cf9c475aa34700f456ef87c1 P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e483d36712e14ba1be32dcacc0b65f4a, State: Initialized, Role: FOLLOWER
I20260927 15:07:25.795689 9475 consensus_queue.cc:260] T 5ae27503cf9c475aa34700f456ef87c1 P e483d36712e14ba1be32dcacc0b65f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:25.795930 9475 ts_tablet_manager.cc:1428] T 5ae27503cf9c475aa34700f456ef87c1 P e483d36712e14ba1be32dcacc0b65f4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 15:07:25.796342 9476 tablet_bootstrap.cc:492] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4: Bootstrap starting.
I20260927 15:07:25.797174 9524 ts_tablet_manager.cc:1905] T 747bd7918a0f45e69927c1002ca8f3b8 P e483d36712e14ba1be32dcacc0b65f4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:25.797358 9476 tablet_bootstrap.cc:654] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:25.799118 9474 tablet_bootstrap.cc:492] T 5ae27503cf9c475aa34700f456ef87c1 P e676da6006f24776bc62de5cd993cee5: Bootstrap starting.
I20260927 15:07:25.800129 9474 tablet_bootstrap.cc:654] T 5ae27503cf9c475aa34700f456ef87c1 P e676da6006f24776bc62de5cd993cee5: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:25.800190 9524 ts_tablet_manager.cc:1918] T 747bd7918a0f45e69927c1002ca8f3b8 P e483d36712e14ba1be32dcacc0b65f4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:07:25.800293 9524 log.cc:1196] T 747bd7918a0f45e69927c1002ca8f3b8 P e483d36712e14ba1be32dcacc0b65f4a: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-1/wal/wals/747bd7918a0f45e69927c1002ca8f3b8
I20260927 15:07:25.800601 9524 ts_tablet_manager.cc:1939] T 747bd7918a0f45e69927c1002ca8f3b8 P e483d36712e14ba1be32dcacc0b65f4a: Deleting consensus metadata
I20260927 15:07:25.801462 9526 ts_tablet_manager.cc:1905] T 747bd7918a0f45e69927c1002ca8f3b8 P 01f11021a9954daea5ab11a7b5e5bbf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:25.801925 8955 catalog_manager.cc:4841] TS e483d36712e14ba1be32dcacc0b65f4a (127.7.17.194:32785): tablet 747bd7918a0f45e69927c1002ca8f3b8 (table test-table [id=528945a8c5a8484eb4261ea543002c1d]) successfully deleted
I20260927 15:07:25.804616 9526 ts_tablet_manager.cc:1918] T 747bd7918a0f45e69927c1002ca8f3b8 P 01f11021a9954daea5ab11a7b5e5bbf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:07:25.804726 9526 log.cc:1196] T 747bd7918a0f45e69927c1002ca8f3b8 P 01f11021a9954daea5ab11a7b5e5bbf4: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/747bd7918a0f45e69927c1002ca8f3b8
I20260927 15:07:25.805048 9526 ts_tablet_manager.cc:1939] T 747bd7918a0f45e69927c1002ca8f3b8 P 01f11021a9954daea5ab11a7b5e5bbf4: Deleting consensus metadata
I20260927 15:07:25.805649 9476 tablet_bootstrap.cc:492] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4: No bootstrap required, opened a new log
I20260927 15:07:25.805744 9476 ts_tablet_manager.cc:1397] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20260927 15:07:25.806224 9476 raft_consensus.cc:348] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:25.806367 9476 raft_consensus.cc:374] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:25.806401 9476 raft_consensus.cc:729] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01f11021a9954daea5ab11a7b5e5bbf4, State: Initialized, Role: FOLLOWER
I20260927 15:07:25.806540 9476 consensus_queue.cc:260] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:25.806833 9476 ts_tablet_manager.cc:1428] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 15:07:25.808177 9474 tablet_bootstrap.cc:492] T 5ae27503cf9c475aa34700f456ef87c1 P e676da6006f24776bc62de5cd993cee5: No bootstrap required, opened a new log
I20260927 15:07:25.808300 9474 ts_tablet_manager.cc:1397] T 5ae27503cf9c475aa34700f456ef87c1 P e676da6006f24776bc62de5cd993cee5: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260927 15:07:25.808781 9474 raft_consensus.cc:348] T 5ae27503cf9c475aa34700f456ef87c1 P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:25.808995 9474 raft_consensus.cc:374] T 5ae27503cf9c475aa34700f456ef87c1 P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:25.809036 9474 raft_consensus.cc:729] T 5ae27503cf9c475aa34700f456ef87c1 P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e676da6006f24776bc62de5cd993cee5, State: Initialized, Role: FOLLOWER
I20260927 15:07:25.809175 9474 consensus_queue.cc:260] T 5ae27503cf9c475aa34700f456ef87c1 P e676da6006f24776bc62de5cd993cee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:25.809480 9474 ts_tablet_manager.cc:1428] T 5ae27503cf9c475aa34700f456ef87c1 P e676da6006f24776bc62de5cd993cee5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:25.811662 8952 catalog_manager.cc:4841] TS 01f11021a9954daea5ab11a7b5e5bbf4 (127.7.17.195:44007): tablet 747bd7918a0f45e69927c1002ca8f3b8 (table test-table [id=528945a8c5a8484eb4261ea543002c1d]) successfully deleted
I20260927 15:07:25.917732 9510 raft_consensus.cc:482] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:25.917898 9510 raft_consensus.cc:504] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:25.918378 9510 leader_election.cc:288] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e676da6006f24776bc62de5cd993cee5 (127.7.17.193:33045), e483d36712e14ba1be32dcacc0b65f4a (127.7.17.194:32785)
I20260927 15:07:25.918960 9125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae27503cf9c475aa34700f456ef87c1" candidate_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e676da6006f24776bc62de5cd993cee5" is_pre_election: true
I20260927 15:07:25.919040 9265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae27503cf9c475aa34700f456ef87c1" candidate_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e483d36712e14ba1be32dcacc0b65f4a" is_pre_election: true
I20260927 15:07:25.919112 9125 raft_consensus.cc:2454] T 5ae27503cf9c475aa34700f456ef87c1 P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f11021a9954daea5ab11a7b5e5bbf4 in term 0.
I20260927 15:07:25.919175 9265 raft_consensus.cc:2454] T 5ae27503cf9c475aa34700f456ef87c1 P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f11021a9954daea5ab11a7b5e5bbf4 in term 0.
I20260927 15:07:25.919550 9339 leader_election.cc:302] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f11021a9954daea5ab11a7b5e5bbf4, e483d36712e14ba1be32dcacc0b65f4a; no voters:
I20260927 15:07:25.919804 9510 raft_consensus.cc:2781] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:25.919878 9510 raft_consensus.cc:482] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:25.919915 9510 raft_consensus.cc:3037] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:25.921260 9510 raft_consensus.cc:504] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:25.921880 9510 leader_election.cc:288] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4 [CANDIDATE]: Term 1 election: Requested vote from peers e676da6006f24776bc62de5cd993cee5 (127.7.17.193:33045), e483d36712e14ba1be32dcacc0b65f4a (127.7.17.194:32785)
I20260927 15:07:25.922376 9125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae27503cf9c475aa34700f456ef87c1" candidate_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e676da6006f24776bc62de5cd993cee5"
I20260927 15:07:25.922415 9265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae27503cf9c475aa34700f456ef87c1" candidate_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e483d36712e14ba1be32dcacc0b65f4a"
I20260927 15:07:25.922497 9125 raft_consensus.cc:3037] T 5ae27503cf9c475aa34700f456ef87c1 P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:25.922595 9265 raft_consensus.cc:3037] T 5ae27503cf9c475aa34700f456ef87c1 P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:25.923609 9125 raft_consensus.cc:2454] T 5ae27503cf9c475aa34700f456ef87c1 P e676da6006f24776bc62de5cd993cee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f11021a9954daea5ab11a7b5e5bbf4 in term 1.
I20260927 15:07:25.923794 9265 raft_consensus.cc:2454] T 5ae27503cf9c475aa34700f456ef87c1 P e483d36712e14ba1be32dcacc0b65f4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f11021a9954daea5ab11a7b5e5bbf4 in term 1.
I20260927 15:07:25.923974 9336 leader_election.cc:302] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f11021a9954daea5ab11a7b5e5bbf4, e676da6006f24776bc62de5cd993cee5; no voters:
I20260927 15:07:25.924177 9510 raft_consensus.cc:2781] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:25.924307 9510 raft_consensus.cc:686] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 1 LEADER]: Becoming Leader. State: Replica: 01f11021a9954daea5ab11a7b5e5bbf4, State: Running, Role: LEADER
I20260927 15:07:25.924454 9510 consensus_queue.cc:237] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:25.926435 8967 catalog_manager.cc:5495] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01f11021a9954daea5ab11a7b5e5bbf4 (127.7.17.195). New cstate: current_term: 1 leader_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } health_report { overall_health: HEALTHY } } }
I20260927 15:07:25.968235 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.965456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33034 (local address 127.7.17.195:44007)
0927 15:07:25.965570 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:25.965574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:25.965585 (+ 11us) server_negotiation.cc:407] Connection header received
0927 15:07:25.965623 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:25.965625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:25.965665 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:25.965729 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:25.965931 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.966464 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.967357 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.967571 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.967649 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.967877 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:25.967945 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:25.967991 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:25.968081 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 15:07:25.968124 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 15:07:25.977691 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.974772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:44007 (local address 127.7.17.195:51827)
0927 15:07:25.974904 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:25.974942 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:25.975033 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:25.975225 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:25.975228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:25.975234 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:07:25.975361 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.975366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.976023 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.976027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.976584 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:25.976591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.977045 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.977049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:25.977147 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.977170 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:07:25.977412 (+ 242us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:07:25.977418 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:07:25.977476 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 15:07:25.977566 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":42}
I20260927 15:07:25.977691 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:25.974868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:51827 (local address 127.7.17.195:44007)
0927 15:07:25.974967 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:25.974971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:25.974984 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:07:25.975115 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:25.975118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:25.975157 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:25.975214 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:25.975444 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.975933 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.976681 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:25.976949 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:25.977034 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:25.977267 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:25.977338 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:25.977402 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:25.977483 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 15:07:25.977536 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":41}
I20260927 15:07:25.983628 9265 raft_consensus.cc:1261] T 5ae27503cf9c475aa34700f456ef87c1 P e483d36712e14ba1be32dcacc0b65f4a [term 1 FOLLOWER]: Refusing update from remote peer 01f11021a9954daea5ab11a7b5e5bbf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:25.983635 9125 raft_consensus.cc:1261] T 5ae27503cf9c475aa34700f456ef87c1 P e676da6006f24776bc62de5cd993cee5 [term 1 FOLLOWER]: Refusing update from remote peer 01f11021a9954daea5ab11a7b5e5bbf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:25.983969 9498 consensus_queue.cc:1035] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:25.984125 9482 consensus_queue.cc:1035] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:25.986549 9503 mvcc.cc:205] Tried to move back new op lower bound from 7333976661945937920 to 7333976661708070912. Current Snapshot: MvccSnapshot[applied={T|T < 7333976661945937920}]
W20260927 15:07:25.997112 9503 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 15:07:27.554394 8968 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:27.554574 8968 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:27.557055 8968 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1: Sending DeleteTablet for 3 replicas of tablet 5ae27503cf9c475aa34700f456ef87c1
I20260927 15:07:27.557638 9105 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae27503cf9c475aa34700f456ef87c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:27 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:35472
I20260927 15:07:27.557667 9245 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae27503cf9c475aa34700f456ef87c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:27 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:39998
I20260927 15:07:27.557780 9384 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae27503cf9c475aa34700f456ef87c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:27 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:33016
I20260927 15:07:27.558012 9556 tablet_replica.cc:330] T 5ae27503cf9c475aa34700f456ef87c1 P e483d36712e14ba1be32dcacc0b65f4a: stopping tablet replica
I20260927 15:07:27.558234 9555 tablet_replica.cc:330] T 5ae27503cf9c475aa34700f456ef87c1 P e676da6006f24776bc62de5cd993cee5: stopping tablet replica
I20260927 15:07:27.558264 9556 raft_consensus.cc:2229] T 5ae27503cf9c475aa34700f456ef87c1 P e483d36712e14ba1be32dcacc0b65f4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:27.558393 9556 raft_consensus.cc:2258] T 5ae27503cf9c475aa34700f456ef87c1 P e483d36712e14ba1be32dcacc0b65f4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:27.558482 9555 raft_consensus.cc:2229] T 5ae27503cf9c475aa34700f456ef87c1 P e676da6006f24776bc62de5cd993cee5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:27.558645 9555 raft_consensus.cc:2258] T 5ae27503cf9c475aa34700f456ef87c1 P e676da6006f24776bc62de5cd993cee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:27.559331 9555 ts_tablet_manager.cc:1905] T 5ae27503cf9c475aa34700f456ef87c1 P e676da6006f24776bc62de5cd993cee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:27.562310 9555 ts_tablet_manager.cc:1918] T 5ae27503cf9c475aa34700f456ef87c1 P e676da6006f24776bc62de5cd993cee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:07:27.562399 9555 log.cc:1196] T 5ae27503cf9c475aa34700f456ef87c1 P e676da6006f24776bc62de5cd993cee5: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-0/wal/wals/5ae27503cf9c475aa34700f456ef87c1
I20260927 15:07:27.562639 9555 ts_tablet_manager.cc:1939] T 5ae27503cf9c475aa34700f456ef87c1 P e676da6006f24776bc62de5cd993cee5: Deleting consensus metadata
I20260927 15:07:27.562971 9556 ts_tablet_manager.cc:1905] T 5ae27503cf9c475aa34700f456ef87c1 P e483d36712e14ba1be32dcacc0b65f4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:27.563431 9557 tablet_replica.cc:330] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4: stopping tablet replica
I20260927 15:07:27.563495 8952 catalog_manager.cc:4841] TS e676da6006f24776bc62de5cd993cee5 (127.7.17.193:33045): tablet 5ae27503cf9c475aa34700f456ef87c1 (table test-table [id=5b0b5f9e74f54be488bb17164e13f3f5]) successfully deleted
I20260927 15:07:27.563570 9557 raft_consensus.cc:2229] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:07:27.563944 9557 raft_consensus.cc:2258] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:27.564512 9557 ts_tablet_manager.cc:1905] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:27.565828 9556 ts_tablet_manager.cc:1918] T 5ae27503cf9c475aa34700f456ef87c1 P e483d36712e14ba1be32dcacc0b65f4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:07:27.565900 9556 log.cc:1196] T 5ae27503cf9c475aa34700f456ef87c1 P e483d36712e14ba1be32dcacc0b65f4a: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-1/wal/wals/5ae27503cf9c475aa34700f456ef87c1
I20260927 15:07:27.566125 9556 ts_tablet_manager.cc:1939] T 5ae27503cf9c475aa34700f456ef87c1 P e483d36712e14ba1be32dcacc0b65f4a: Deleting consensus metadata
I20260927 15:07:27.566810 8955 catalog_manager.cc:4841] TS e483d36712e14ba1be32dcacc0b65f4a (127.7.17.194:32785): tablet 5ae27503cf9c475aa34700f456ef87c1 (table test-table [id=5b0b5f9e74f54be488bb17164e13f3f5]) successfully deleted
I20260927 15:07:27.566932 9557 ts_tablet_manager.cc:1918] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:07:27.567010 9557 log.cc:1196] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/5ae27503cf9c475aa34700f456ef87c1
I20260927 15:07:27.567250 9557 ts_tablet_manager.cc:1939] T 5ae27503cf9c475aa34700f456ef87c1 P 01f11021a9954daea5ab11a7b5e5bbf4: Deleting consensus metadata
I20260927 15:07:27.567900 8952 catalog_manager.cc:4841] TS 01f11021a9954daea5ab11a7b5e5bbf4 (127.7.17.195:44007): tablet 5ae27503cf9c475aa34700f456ef87c1 (table test-table [id=5b0b5f9e74f54be488bb17164e13f3f5]) successfully deleted
Sep 27 15:07:27 dist-test-slave-hhpq krb5kdc[8923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521647, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:07:27 dist-test-slave-hhpq krb5kdc[8923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521647, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:27.591609 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:27.577579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32938 (local address 127.7.17.254:40013)
0927 15:07:27.577853 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:27.577860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:27.577877 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:07:27.578003 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:27.578006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:27.578053 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:27.578159 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:27.578840 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:27.579522 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:27.580358 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:27.580611 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:27.580685 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:27.588683 (+ 7998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:27.588687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:27.588689 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:27.588723 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:27.590187 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:27.590613 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:27.590616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:27.590618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:27.590698 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:27.590902 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:27.590904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:27.590905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:27.591085 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:27.591183 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:27.591361 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 15:07:27.591473 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":105,"threads_started":1}
I20260927 15:07:27.594293 8968 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:07:27.594743 8968 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:27.599427 9105 tablet_service.cc:1467] Processing CreateTablet for tablet 5c847ed3939a48bbacb40e4d20d81958 (DEFAULT_TABLE table=test-table [id=7e2671a46c6d42e2a24d075e1a7ac636]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:27.599510 9245 tablet_service.cc:1467] Processing CreateTablet for tablet 5c847ed3939a48bbacb40e4d20d81958 (DEFAULT_TABLE table=test-table [id=7e2671a46c6d42e2a24d075e1a7ac636]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:27.599622 9384 tablet_service.cc:1467] Processing CreateTablet for tablet 5c847ed3939a48bbacb40e4d20d81958 (DEFAULT_TABLE table=test-table [id=7e2671a46c6d42e2a24d075e1a7ac636]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:27.599753 9105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c847ed3939a48bbacb40e4d20d81958. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:27.599787 9245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c847ed3939a48bbacb40e4d20d81958. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:27.599860 9384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c847ed3939a48bbacb40e4d20d81958. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:27.601799 9566 tablet_bootstrap.cc:492] T 5c847ed3939a48bbacb40e4d20d81958 P e483d36712e14ba1be32dcacc0b65f4a: Bootstrap starting.
I20260927 15:07:27.602303 9567 tablet_bootstrap.cc:492] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5: Bootstrap starting.
I20260927 15:07:27.602694 9568 tablet_bootstrap.cc:492] T 5c847ed3939a48bbacb40e4d20d81958 P 01f11021a9954daea5ab11a7b5e5bbf4: Bootstrap starting.
I20260927 15:07:27.603101 9566 tablet_bootstrap.cc:654] T 5c847ed3939a48bbacb40e4d20d81958 P e483d36712e14ba1be32dcacc0b65f4a: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:27.603140 9567 tablet_bootstrap.cc:654] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:27.603523 9568 tablet_bootstrap.cc:654] T 5c847ed3939a48bbacb40e4d20d81958 P 01f11021a9954daea5ab11a7b5e5bbf4: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:27.604216 9567 tablet_bootstrap.cc:492] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5: No bootstrap required, opened a new log
I20260927 15:07:27.604302 9567 ts_tablet_manager.cc:1397] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:07:27.604578 9568 tablet_bootstrap.cc:492] T 5c847ed3939a48bbacb40e4d20d81958 P 01f11021a9954daea5ab11a7b5e5bbf4: No bootstrap required, opened a new log
I20260927 15:07:27.604656 9568 ts_tablet_manager.cc:1397] T 5c847ed3939a48bbacb40e4d20d81958 P 01f11021a9954daea5ab11a7b5e5bbf4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:07:27.604717 9567 raft_consensus.cc:348] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:27.604864 9567 raft_consensus.cc:374] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:27.604904 9567 raft_consensus.cc:729] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e676da6006f24776bc62de5cd993cee5, State: Initialized, Role: FOLLOWER
I20260927 15:07:27.605031 9567 consensus_queue.cc:260] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:27.605067 9568 raft_consensus.cc:348] T 5c847ed3939a48bbacb40e4d20d81958 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:27.605176 9568 raft_consensus.cc:374] T 5c847ed3939a48bbacb40e4d20d81958 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:27.605211 9568 raft_consensus.cc:729] T 5c847ed3939a48bbacb40e4d20d81958 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01f11021a9954daea5ab11a7b5e5bbf4, State: Initialized, Role: FOLLOWER
I20260927 15:07:27.605325 9568 consensus_queue.cc:260] T 5c847ed3939a48bbacb40e4d20d81958 P 01f11021a9954daea5ab11a7b5e5bbf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:27.605400 9567 ts_tablet_manager.cc:1428] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 15:07:27.605600 9568 ts_tablet_manager.cc:1428] T 5c847ed3939a48bbacb40e4d20d81958 P 01f11021a9954daea5ab11a7b5e5bbf4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:27.605959 9566 tablet_bootstrap.cc:492] T 5c847ed3939a48bbacb40e4d20d81958 P e483d36712e14ba1be32dcacc0b65f4a: No bootstrap required, opened a new log
I20260927 15:07:27.606050 9566 ts_tablet_manager.cc:1397] T 5c847ed3939a48bbacb40e4d20d81958 P e483d36712e14ba1be32dcacc0b65f4a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 15:07:27.606427 9566 raft_consensus.cc:348] T 5c847ed3939a48bbacb40e4d20d81958 P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:27.606520 9566 raft_consensus.cc:374] T 5c847ed3939a48bbacb40e4d20d81958 P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:27.606549 9566 raft_consensus.cc:729] T 5c847ed3939a48bbacb40e4d20d81958 P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e483d36712e14ba1be32dcacc0b65f4a, State: Initialized, Role: FOLLOWER
I20260927 15:07:27.606678 9566 consensus_queue.cc:260] T 5c847ed3939a48bbacb40e4d20d81958 P e483d36712e14ba1be32dcacc0b65f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:27.607084 9566 ts_tablet_manager.cc:1428] T 5c847ed3939a48bbacb40e4d20d81958 P e483d36712e14ba1be32dcacc0b65f4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:27.783638 9572 raft_consensus.cc:482] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:27.783803 9572 raft_consensus.cc:504] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:27.784237 9572 leader_election.cc:288] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e483d36712e14ba1be32dcacc0b65f4a (127.7.17.194:32785), 01f11021a9954daea5ab11a7b5e5bbf4 (127.7.17.195:44007)
I20260927 15:07:27.784757 9405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c847ed3939a48bbacb40e4d20d81958" candidate_uuid: "e676da6006f24776bc62de5cd993cee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" is_pre_election: true
I20260927 15:07:27.784759 9265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c847ed3939a48bbacb40e4d20d81958" candidate_uuid: "e676da6006f24776bc62de5cd993cee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e483d36712e14ba1be32dcacc0b65f4a" is_pre_election: true
I20260927 15:07:27.784920 9405 raft_consensus.cc:2454] T 5c847ed3939a48bbacb40e4d20d81958 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e676da6006f24776bc62de5cd993cee5 in term 0.
I20260927 15:07:27.784934 9265 raft_consensus.cc:2454] T 5c847ed3939a48bbacb40e4d20d81958 P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e676da6006f24776bc62de5cd993cee5 in term 0.
I20260927 15:07:27.785261 9056 leader_election.cc:302] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 01f11021a9954daea5ab11a7b5e5bbf4, e483d36712e14ba1be32dcacc0b65f4a, e676da6006f24776bc62de5cd993cee5; no voters:
I20260927 15:07:27.785463 9572 raft_consensus.cc:2781] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:27.785529 9572 raft_consensus.cc:482] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:27.785575 9572 raft_consensus.cc:3037] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:27.786649 9572 raft_consensus.cc:504] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:27.786999 9572 leader_election.cc:288] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5 [CANDIDATE]: Term 1 election: Requested vote from peers e483d36712e14ba1be32dcacc0b65f4a (127.7.17.194:32785), 01f11021a9954daea5ab11a7b5e5bbf4 (127.7.17.195:44007)
I20260927 15:07:27.787443 9405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c847ed3939a48bbacb40e4d20d81958" candidate_uuid: "e676da6006f24776bc62de5cd993cee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01f11021a9954daea5ab11a7b5e5bbf4"
I20260927 15:07:27.787444 9265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c847ed3939a48bbacb40e4d20d81958" candidate_uuid: "e676da6006f24776bc62de5cd993cee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e483d36712e14ba1be32dcacc0b65f4a"
I20260927 15:07:27.787588 9265 raft_consensus.cc:3037] T 5c847ed3939a48bbacb40e4d20d81958 P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:27.787588 9405 raft_consensus.cc:3037] T 5c847ed3939a48bbacb40e4d20d81958 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:27.788777 9405 raft_consensus.cc:2454] T 5c847ed3939a48bbacb40e4d20d81958 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e676da6006f24776bc62de5cd993cee5 in term 1.
I20260927 15:07:27.788772 9265 raft_consensus.cc:2454] T 5c847ed3939a48bbacb40e4d20d81958 P e483d36712e14ba1be32dcacc0b65f4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e676da6006f24776bc62de5cd993cee5 in term 1.
I20260927 15:07:27.789122 9056 leader_election.cc:302] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f11021a9954daea5ab11a7b5e5bbf4, e676da6006f24776bc62de5cd993cee5; no voters:
I20260927 15:07:27.789323 9572 raft_consensus.cc:2781] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:27.789552 9572 raft_consensus.cc:686] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5 [term 1 LEADER]: Becoming Leader. State: Replica: e676da6006f24776bc62de5cd993cee5, State: Running, Role: LEADER
I20260927 15:07:27.789697 9572 consensus_queue.cc:237] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:27.791370 8968 catalog_manager.cc:5495] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e676da6006f24776bc62de5cd993cee5 (127.7.17.193). New cstate: current_term: 1 leader_uuid: "e676da6006f24776bc62de5cd993cee5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } health_report { overall_health: UNKNOWN } } }
I20260927 15:07:27.851444 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:27.848334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35492 (local address 127.7.17.193:33045)
0927 15:07:27.848594 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:27.848600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:27.848613 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:07:27.848652 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:27.848654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:27.848695 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:27.848784 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:27.849078 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:27.849629 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:27.850552 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:27.850770 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:27.850840 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:27.851075 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:27.851147 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:27.851195 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:27.851257 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 15:07:27.851303 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":97,"threads_started":1}
I20260927 15:07:27.857388 9405 raft_consensus.cc:1261] T 5c847ed3939a48bbacb40e4d20d81958 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 1 FOLLOWER]: Refusing update from remote peer e676da6006f24776bc62de5cd993cee5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:27.857447 9265 raft_consensus.cc:1261] T 5c847ed3939a48bbacb40e4d20d81958 P e483d36712e14ba1be32dcacc0b65f4a [term 1 FOLLOWER]: Refusing update from remote peer e676da6006f24776bc62de5cd993cee5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:27.857837 9574 consensus_queue.cc:1035] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:27.858011 9572 consensus_queue.cc:1035] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:27.885532 8966 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:27.885692 8966 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:27.887543 8966 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1: Sending DeleteTablet for 3 replicas of tablet 5c847ed3939a48bbacb40e4d20d81958
I20260927 15:07:27.888034 9105 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c847ed3939a48bbacb40e4d20d81958 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:27 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:35472
I20260927 15:07:27.888095 9245 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c847ed3939a48bbacb40e4d20d81958 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:27 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:39998
I20260927 15:07:27.888108 9384 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c847ed3939a48bbacb40e4d20d81958 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:27 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:33016
I20260927 15:07:27.888211 9555 tablet_replica.cc:330] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5: stopping tablet replica
I20260927 15:07:27.888242 9556 tablet_replica.cc:330] T 5c847ed3939a48bbacb40e4d20d81958 P e483d36712e14ba1be32dcacc0b65f4a: stopping tablet replica
I20260927 15:07:27.888324 9556 raft_consensus.cc:2229] T 5c847ed3939a48bbacb40e4d20d81958 P e483d36712e14ba1be32dcacc0b65f4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:27.888320 9555 raft_consensus.cc:2229] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:07:27.888476 9556 raft_consensus.cc:2258] T 5c847ed3939a48bbacb40e4d20d81958 P e483d36712e14ba1be32dcacc0b65f4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:27.888507 9555 raft_consensus.cc:2258] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:27.888655 9557 tablet_replica.cc:330] T 5c847ed3939a48bbacb40e4d20d81958 P 01f11021a9954daea5ab11a7b5e5bbf4: stopping tablet replica
I20260927 15:07:27.888769 9557 raft_consensus.cc:2229] T 5c847ed3939a48bbacb40e4d20d81958 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:27.888916 9557 raft_consensus.cc:2258] T 5c847ed3939a48bbacb40e4d20d81958 P 01f11021a9954daea5ab11a7b5e5bbf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:27.889065 8966 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 15:07:27.889222 9555 ts_tablet_manager.cc:1905] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 15:07:27.889503 8966 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:27.889824 9556 ts_tablet_manager.cc:1905] T 5c847ed3939a48bbacb40e4d20d81958 P e483d36712e14ba1be32dcacc0b65f4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:27.891789 9555 ts_tablet_manager.cc:1918] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:07:27.891868 9555 log.cc:1196] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-0/wal/wals/5c847ed3939a48bbacb40e4d20d81958
I20260927 15:07:27.892117 9555 ts_tablet_manager.cc:1939] T 5c847ed3939a48bbacb40e4d20d81958 P e676da6006f24776bc62de5cd993cee5: Deleting consensus metadata
I20260927 15:07:27.892139 9556 ts_tablet_manager.cc:1918] T 5c847ed3939a48bbacb40e4d20d81958 P e483d36712e14ba1be32dcacc0b65f4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:07:27.892195 9556 log.cc:1196] T 5c847ed3939a48bbacb40e4d20d81958 P e483d36712e14ba1be32dcacc0b65f4a: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-1/wal/wals/5c847ed3939a48bbacb40e4d20d81958
I20260927 15:07:27.892408 9556 ts_tablet_manager.cc:1939] T 5c847ed3939a48bbacb40e4d20d81958 P e483d36712e14ba1be32dcacc0b65f4a: Deleting consensus metadata
I20260927 15:07:27.892812 8952 catalog_manager.cc:4841] TS e676da6006f24776bc62de5cd993cee5 (127.7.17.193:33045): tablet 5c847ed3939a48bbacb40e4d20d81958 (table test-table [id=7e2671a46c6d42e2a24d075e1a7ac636]) successfully deleted
I20260927 15:07:27.893146 9557 ts_tablet_manager.cc:1905] T 5c847ed3939a48bbacb40e4d20d81958 P 01f11021a9954daea5ab11a7b5e5bbf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:27.893312 8955 catalog_manager.cc:4841] TS e483d36712e14ba1be32dcacc0b65f4a (127.7.17.194:32785): tablet 5c847ed3939a48bbacb40e4d20d81958 (table test-table [id=7e2671a46c6d42e2a24d075e1a7ac636]) successfully deleted
I20260927 15:07:27.894212 9105 tablet_service.cc:1467] Processing CreateTablet for tablet f57b6b14684d455ca2bfc976ad2813ca (DEFAULT_TABLE table=test-table [id=7cbfaff4d42c4e14b2b3efea581a4e07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:27.894456 9105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f57b6b14684d455ca2bfc976ad2813ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:27.894675 9384 tablet_service.cc:1467] Processing CreateTablet for tablet f57b6b14684d455ca2bfc976ad2813ca (DEFAULT_TABLE table=test-table [id=7cbfaff4d42c4e14b2b3efea581a4e07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:27.894914 9384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f57b6b14684d455ca2bfc976ad2813ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:27.895646 9245 tablet_service.cc:1467] Processing CreateTablet for tablet f57b6b14684d455ca2bfc976ad2813ca (DEFAULT_TABLE table=test-table [id=7cbfaff4d42c4e14b2b3efea581a4e07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:27.895884 9245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f57b6b14684d455ca2bfc976ad2813ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:27.896639 9567 tablet_bootstrap.cc:492] T f57b6b14684d455ca2bfc976ad2813ca P e676da6006f24776bc62de5cd993cee5: Bootstrap starting.
I20260927 15:07:27.896901 9568 tablet_bootstrap.cc:492] T f57b6b14684d455ca2bfc976ad2813ca P 01f11021a9954daea5ab11a7b5e5bbf4: Bootstrap starting.
I20260927 15:07:27.897563 9567 tablet_bootstrap.cc:654] T f57b6b14684d455ca2bfc976ad2813ca P e676da6006f24776bc62de5cd993cee5: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:27.897589 9568 tablet_bootstrap.cc:654] T f57b6b14684d455ca2bfc976ad2813ca P 01f11021a9954daea5ab11a7b5e5bbf4: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:27.897850 9566 tablet_bootstrap.cc:492] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a: Bootstrap starting.
I20260927 15:07:27.898139 9557 ts_tablet_manager.cc:1918] T 5c847ed3939a48bbacb40e4d20d81958 P 01f11021a9954daea5ab11a7b5e5bbf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:07:27.898228 9557 log.cc:1196] T 5c847ed3939a48bbacb40e4d20d81958 P 01f11021a9954daea5ab11a7b5e5bbf4: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/5c847ed3939a48bbacb40e4d20d81958
I20260927 15:07:27.898501 9557 ts_tablet_manager.cc:1939] T 5c847ed3939a48bbacb40e4d20d81958 P 01f11021a9954daea5ab11a7b5e5bbf4: Deleting consensus metadata
I20260927 15:07:27.898767 9566 tablet_bootstrap.cc:654] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:27.899180 8952 catalog_manager.cc:4841] TS 01f11021a9954daea5ab11a7b5e5bbf4 (127.7.17.195:44007): tablet 5c847ed3939a48bbacb40e4d20d81958 (table test-table [id=7e2671a46c6d42e2a24d075e1a7ac636]) successfully deleted
I20260927 15:07:27.899766 9568 tablet_bootstrap.cc:492] T f57b6b14684d455ca2bfc976ad2813ca P 01f11021a9954daea5ab11a7b5e5bbf4: No bootstrap required, opened a new log
I20260927 15:07:27.899848 9568 ts_tablet_manager.cc:1397] T f57b6b14684d455ca2bfc976ad2813ca P 01f11021a9954daea5ab11a7b5e5bbf4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 15:07:27.900000 9566 tablet_bootstrap.cc:492] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a: No bootstrap required, opened a new log
I20260927 15:07:27.900007 9567 tablet_bootstrap.cc:492] T f57b6b14684d455ca2bfc976ad2813ca P e676da6006f24776bc62de5cd993cee5: No bootstrap required, opened a new log
I20260927 15:07:27.900089 9566 ts_tablet_manager.cc:1397] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:07:27.900089 9567 ts_tablet_manager.cc:1397] T f57b6b14684d455ca2bfc976ad2813ca P e676da6006f24776bc62de5cd993cee5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 15:07:27.900242 9568 raft_consensus.cc:348] T f57b6b14684d455ca2bfc976ad2813ca P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:27.900341 9568 raft_consensus.cc:374] T f57b6b14684d455ca2bfc976ad2813ca P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:27.900393 9568 raft_consensus.cc:729] T f57b6b14684d455ca2bfc976ad2813ca P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01f11021a9954daea5ab11a7b5e5bbf4, State: Initialized, Role: FOLLOWER
I20260927 15:07:27.900497 9567 raft_consensus.cc:348] T f57b6b14684d455ca2bfc976ad2813ca P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:27.900516 9568 consensus_queue.cc:260] T f57b6b14684d455ca2bfc976ad2813ca P 01f11021a9954daea5ab11a7b5e5bbf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:27.900488 9566 raft_consensus.cc:348] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:27.900606 9566 raft_consensus.cc:374] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:27.900606 9567 raft_consensus.cc:374] T f57b6b14684d455ca2bfc976ad2813ca P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:27.900647 9567 raft_consensus.cc:729] T f57b6b14684d455ca2bfc976ad2813ca P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e676da6006f24776bc62de5cd993cee5, State: Initialized, Role: FOLLOWER
I20260927 15:07:27.900647 9566 raft_consensus.cc:729] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e483d36712e14ba1be32dcacc0b65f4a, State: Initialized, Role: FOLLOWER
I20260927 15:07:27.900776 9568 ts_tablet_manager.cc:1428] T f57b6b14684d455ca2bfc976ad2813ca P 01f11021a9954daea5ab11a7b5e5bbf4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:27.900761 9567 consensus_queue.cc:260] T f57b6b14684d455ca2bfc976ad2813ca P e676da6006f24776bc62de5cd993cee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:27.900761 9566 consensus_queue.cc:260] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:27.901031 9566 ts_tablet_manager.cc:1428] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:27.901039 9567 ts_tablet_manager.cc:1428] T f57b6b14684d455ca2bfc976ad2813ca P e676da6006f24776bc62de5cd993cee5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:27.997888 9573 raft_consensus.cc:482] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:27.998025 9573 raft_consensus.cc:504] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:27.999125 9573 leader_election.cc:288] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e676da6006f24776bc62de5cd993cee5 (127.7.17.193:33045), 01f11021a9954daea5ab11a7b5e5bbf4 (127.7.17.195:44007)
I20260927 15:07:28.003901 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:27.999451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:59827 (local address 127.7.17.193:33045)
0927 15:07:27.999578 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:27.999582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:27.999732 (+ 150us) server_negotiation.cc:407] Connection header received
0927 15:07:27.999830 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:27.999833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:27.999883 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:27.999971 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:28.000279 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:28.001076 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:28.002647 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:28.003262 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:28.003391 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:28.003436 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:28.003697 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 15:07:28.003767 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:07:28.003926 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:27.999804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:47641 (local address 127.7.17.195:44007)
0927 15:07:28.000088 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:28.000092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:28.000105 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:07:28.000175 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:28.000177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:28.000212 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:28.000299 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:28.000515 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:28.001306 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:28.002647 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:28.003266 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:28.003383 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:28.003436 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:28.003697 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 15:07:28.003767 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":102,"threads_started":1}
I20260927 15:07:28.003957 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:27.999706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:44007 (local address 127.7.17.194:47641)
0927 15:07:28.000018 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:28.000030 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:28.000101 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:28.000322 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:28.000324 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:28.000337 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:28.000417 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:28.000420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:28.001487 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:28.001491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:28.002557 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:28.002564 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:28.003385 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:28.003389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:28.003564 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:28.003600 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:07:28.003677 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 15:07:28.003780 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":69,"thread_start_us":100,"threads_started":1}
I20260927 15:07:28.004426 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:27.999354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:33045 (local address 127.7.17.194:59827)
0927 15:07:27.999596 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:27.999611 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:27.999732 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:28.000000 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:28.000003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:28.000023 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:28.000174 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:28.000181 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:28.001191 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:28.001194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:28.002516 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:28.002527 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:28.003385 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:28.003389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:28.003554 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:28.003606 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 15:07:28.003677 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 15:07:28.003779 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":71,"thread_start_us":81,"threads_started":1}
I20260927 15:07:28.004376 9405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57b6b14684d455ca2bfc976ad2813ca" candidate_uuid: "e483d36712e14ba1be32dcacc0b65f4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" is_pre_election: true
I20260927 15:07:28.004536 9405 raft_consensus.cc:2454] T f57b6b14684d455ca2bfc976ad2813ca P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e483d36712e14ba1be32dcacc0b65f4a in term 0.
I20260927 15:07:28.004884 9124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57b6b14684d455ca2bfc976ad2813ca" candidate_uuid: "e483d36712e14ba1be32dcacc0b65f4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e676da6006f24776bc62de5cd993cee5" is_pre_election: true
I20260927 15:07:28.004915 9196 leader_election.cc:302] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f11021a9954daea5ab11a7b5e5bbf4, e483d36712e14ba1be32dcacc0b65f4a; no voters:
I20260927 15:07:28.005014 9124 raft_consensus.cc:2454] T f57b6b14684d455ca2bfc976ad2813ca P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e483d36712e14ba1be32dcacc0b65f4a in term 0.
I20260927 15:07:28.005164 9573 raft_consensus.cc:2781] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:28.005227 9573 raft_consensus.cc:482] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:28.005287 9573 raft_consensus.cc:3037] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:28.006264 9573 raft_consensus.cc:504] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:28.006605 9573 leader_election.cc:288] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a [CANDIDATE]: Term 1 election: Requested vote from peers e676da6006f24776bc62de5cd993cee5 (127.7.17.193:33045), 01f11021a9954daea5ab11a7b5e5bbf4 (127.7.17.195:44007)
I20260927 15:07:28.007018 9124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57b6b14684d455ca2bfc976ad2813ca" candidate_uuid: "e483d36712e14ba1be32dcacc0b65f4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e676da6006f24776bc62de5cd993cee5"
I20260927 15:07:28.007045 9405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57b6b14684d455ca2bfc976ad2813ca" candidate_uuid: "e483d36712e14ba1be32dcacc0b65f4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01f11021a9954daea5ab11a7b5e5bbf4"
I20260927 15:07:28.007133 9124 raft_consensus.cc:3037] T f57b6b14684d455ca2bfc976ad2813ca P e676da6006f24776bc62de5cd993cee5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:28.007144 9405 raft_consensus.cc:3037] T f57b6b14684d455ca2bfc976ad2813ca P 01f11021a9954daea5ab11a7b5e5bbf4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:28.008080 9124 raft_consensus.cc:2454] T f57b6b14684d455ca2bfc976ad2813ca P e676da6006f24776bc62de5cd993cee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e483d36712e14ba1be32dcacc0b65f4a in term 1.
I20260927 15:07:28.008083 9405 raft_consensus.cc:2454] T f57b6b14684d455ca2bfc976ad2813ca P 01f11021a9954daea5ab11a7b5e5bbf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e483d36712e14ba1be32dcacc0b65f4a in term 1.
I20260927 15:07:28.008333 9196 leader_election.cc:302] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f11021a9954daea5ab11a7b5e5bbf4, e483d36712e14ba1be32dcacc0b65f4a; no voters:
I20260927 15:07:28.008512 9573 raft_consensus.cc:2781] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:28.008754 9573 raft_consensus.cc:686] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a [term 1 LEADER]: Becoming Leader. State: Replica: e483d36712e14ba1be32dcacc0b65f4a, State: Running, Role: LEADER
I20260927 15:07:28.009043 9573 consensus_queue.cc:237] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } }
I20260927 15:07:28.011725 8967 catalog_manager.cc:5495] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a reported cstate change: term changed from 0 to 1, leader changed from <none> to e483d36712e14ba1be32dcacc0b65f4a (127.7.17.194). New cstate: current_term: 1 leader_uuid: "e483d36712e14ba1be32dcacc0b65f4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e483d36712e14ba1be32dcacc0b65f4a" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 32785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 } health_report { overall_health: UNKNOWN } } }
I20260927 15:07:28.031735 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:28.027943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40004 (local address 127.7.17.194:32785)
0927 15:07:28.028274 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:28.028281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:28.028292 (+ 11us) server_negotiation.cc:407] Connection header received
0927 15:07:28.028362 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:28.028364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:28.028416 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:28.028523 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:28.028720 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:28.029411 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:28.030298 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:28.030570 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:28.030648 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:28.030862 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:28.031327 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:28.031410 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:28.031554 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 15:07:28.031604 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":179,"threads_started":1}
I20260927 15:07:28.041968 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:28.038944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:55183 (local address 127.7.17.195:44007)
0927 15:07:28.039059 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:28.039063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:28.039121 (+ 58us) server_negotiation.cc:407] Connection header received
0927 15:07:28.039236 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:28.039238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:28.039283 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:28.039347 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:28.039537 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:28.040047 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:28.040955 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:28.041163 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:28.041236 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:28.041554 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:28.041627 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:28.041679 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:28.041781 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 15:07:28.041831 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:07:28.042140 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:28.038749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:44007 (local address 127.7.17.194:55183)
0927 15:07:28.039045 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:28.039059 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:28.039157 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:28.039358 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:28.039360 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:28.039365 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:07:28.039464 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:28.039468 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:28.040169 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:28.040172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:28.040817 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:28.040856 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:28.041287 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:28.041291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:28.041365 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:28.041444 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:07:28.041710 (+ 266us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:07:28.041715 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:07:28.041768 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 15:07:28.041836 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":95,"threads_started":1}
I20260927 15:07:28.048040 9124 raft_consensus.cc:1261] T f57b6b14684d455ca2bfc976ad2813ca P e676da6006f24776bc62de5cd993cee5 [term 1 FOLLOWER]: Refusing update from remote peer e483d36712e14ba1be32dcacc0b65f4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:28.048032 9405 raft_consensus.cc:1261] T f57b6b14684d455ca2bfc976ad2813ca P 01f11021a9954daea5ab11a7b5e5bbf4 [term 1 FOLLOWER]: Refusing update from remote peer e483d36712e14ba1be32dcacc0b65f4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:28.048496 9573 consensus_queue.cc:1035] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e676da6006f24776bc62de5cd993cee5" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:28.048695 9588 consensus_queue.cc:1035] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f11021a9954daea5ab11a7b5e5bbf4" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:28.055166 9540 mvcc.cc:205] Tried to move back new op lower bound from 7333976670400434176 to 7333976670247886848. Current Snapshot: MvccSnapshot[applied={T|T < 7333976670400434176}]
I20260927 15:07:28.066509 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:28.063065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:44911 (local address 127.7.17.194:32785)
0927 15:07:28.063164 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:28.063168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:28.063398 (+ 230us) server_negotiation.cc:407] Connection header received
0927 15:07:28.063503 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:28.063506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:28.063549 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:28.063606 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:28.063825 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:28.064407 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:28.065467 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:28.065691 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:28.065782 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:28.066106 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:28.066179 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:28.066233 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:28.066327 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 15:07:28.066376 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 15:07:28.066524 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:28.062997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:32785 (local address 127.7.17.195:44911)
0927 15:07:28.063311 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:28.063326 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:28.063425 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:28.063619 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:28.063622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:28.063629 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:07:28.063726 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:28.063732 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:28.064501 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:28.064505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:28.065364 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:28.065375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:28.065908 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:28.065911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:28.065977 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:28.066002 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:07:28.066254 (+ 252us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:07:28.066260 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:07:28.066318 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 15:07:28.066399 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":101,"threads_started":1}
W20260927 15:07:28.067155 9539 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 15:07:29.623500 8968 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:29.623654 8968 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:29.625999 8968 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 136271aab3344dc0b0ae2ff9f45bd0d1: Sending DeleteTablet for 3 replicas of tablet f57b6b14684d455ca2bfc976ad2813ca
I20260927 15:07:29.626559 9245 tablet_service.cc:1514] Processing DeleteTablet for tablet f57b6b14684d455ca2bfc976ad2813ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:29 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:39998
I20260927 15:07:29.626576 9105 tablet_service.cc:1514] Processing DeleteTablet for tablet f57b6b14684d455ca2bfc976ad2813ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:29 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:35472
I20260927 15:07:29.626571 9384 tablet_service.cc:1514] Processing DeleteTablet for tablet f57b6b14684d455ca2bfc976ad2813ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:29 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:33016
I20260927 15:07:29.626937 9624 tablet_replica.cc:330] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a: stopping tablet replica
I20260927 15:07:29.627045 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 9040
I20260927 15:07:29.627060 9624 raft_consensus.cc:2229] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:07:29.627871 9624 raft_consensus.cc:2258] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:29.628142 9622 tablet_replica.cc:330] T f57b6b14684d455ca2bfc976ad2813ca P 01f11021a9954daea5ab11a7b5e5bbf4: stopping tablet replica
I20260927 15:07:29.628271 9622 raft_consensus.cc:2229] T f57b6b14684d455ca2bfc976ad2813ca P 01f11021a9954daea5ab11a7b5e5bbf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:29.628398 9622 raft_consensus.cc:2258] T f57b6b14684d455ca2bfc976ad2813ca P 01f11021a9954daea5ab11a7b5e5bbf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:29.628502 9624 ts_tablet_manager.cc:1905] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:29.629338 9622 ts_tablet_manager.cc:1905] T f57b6b14684d455ca2bfc976ad2813ca P 01f11021a9954daea5ab11a7b5e5bbf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:29.631331 9624 ts_tablet_manager.cc:1918] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:07:29.631508 9624 log.cc:1196] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-1/wal/wals/f57b6b14684d455ca2bfc976ad2813ca
I20260927 15:07:29.631776 9624 ts_tablet_manager.cc:1939] T f57b6b14684d455ca2bfc976ad2813ca P e483d36712e14ba1be32dcacc0b65f4a: Deleting consensus metadata
I20260927 15:07:29.631945 9622 ts_tablet_manager.cc:1918] T f57b6b14684d455ca2bfc976ad2813ca P 01f11021a9954daea5ab11a7b5e5bbf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:07:29.632042 9622 log.cc:1196] T f57b6b14684d455ca2bfc976ad2813ca P 01f11021a9954daea5ab11a7b5e5bbf4: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/f57b6b14684d455ca2bfc976ad2813ca
I20260927 15:07:29.632261 9622 ts_tablet_manager.cc:1939] T f57b6b14684d455ca2bfc976ad2813ca P 01f11021a9954daea5ab11a7b5e5bbf4: Deleting consensus metadata
I20260927 15:07:29.632609 8955 catalog_manager.cc:4841] TS e483d36712e14ba1be32dcacc0b65f4a (127.7.17.194:32785): tablet f57b6b14684d455ca2bfc976ad2813ca (table test-table [id=7cbfaff4d42c4e14b2b3efea581a4e07]) successfully deleted
I20260927 15:07:29.632915 8952 catalog_manager.cc:4841] TS 01f11021a9954daea5ab11a7b5e5bbf4 (127.7.17.195:44007): tablet f57b6b14684d455ca2bfc976ad2813ca (table test-table [id=7cbfaff4d42c4e14b2b3efea581a4e07]) successfully deleted
W20260927 15:07:29.635386 8952 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.17.193:33045 (error 108)
I20260927 15:07:29.635529 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 9180
I20260927 15:07:29.637120 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:29.636615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:33045 (local address 127.0.0.1:35506)
0927 15:07:29.636924 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:29.637007 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.193:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":92,"threads_started":1}
W20260927 15:07:29.641026 8952 catalog_manager.cc:4584] TS e676da6006f24776bc62de5cd993cee5 (127.7.17.193:33045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f57b6b14684d455ca2bfc976ad2813ca: Network error: Client connection negotiation failed: client connection to 127.7.17.193:33045: connect: Connection refused (error 111)
I20260927 15:07:29.641939 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 9320
I20260927 15:07:29.647964 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 8936
2026-09-27T15:07:29Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6515 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:07:29 dist-test-slave-hhpq krb5kdc[9629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:07:29 dist-test-slave-hhpq krb5kdc[9629](info): set up 2 sockets
Sep 27 15:07:29 dist-test-slave-hhpq krb5kdc[9629](info): commencing operation
krb5kdc: starting...
W20260927 15:07:30.604552 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:07:30 dist-test-slave-hhpq krb5kdc[9629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:07:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:07:30.728071 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:46879
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:46879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:30.839218 9642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:30.839504 9642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:30.839543 9642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:30.843592 9642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:30.843659 9642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:30.843677 9642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:30.843693 9642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:30.843708 9642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:30.849062 9642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:46879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:46879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:30.850140 9642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:30.851188 9642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:07:30.859913 9642 server_base.cc:1034] running on GCE node
W20260927 15:07:30.860081 9647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:30.859994 9650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:30.860025 9648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:30.860713 9642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:30.861891 9642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:30.863063 9642 hybrid_clock.cc:648] HybridClock initialized: now 1790521650863045 us; error 28 us; skew 500 ppm
Sep 27 15:07:30 dist-test-slave-hhpq krb5kdc[9629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521650, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:30.865989 9642 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:07:30.867017 9642 webserver.cc:460] Webserver started at http://127.7.17.254:46847/ using document root <none> and password file <none>
I20260927 15:07:30.867625 9642 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:30.867691 9642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:30.867892 9642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:30.870044 9642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "1bf59df430f84a1892700c9cb9092e7a"
format_stamp: "Formatted at 2026-09-27 15:07:30 on dist-test-slave-hhpq"
server_key: "3ba40cd01137162db8ab47fbdde4712e"
server_key_iv: "3235eacb60eb17a06aa836caa2164649"
server_key_version: "encryptionkey@0"
I20260927 15:07:30.870608 9642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "1bf59df430f84a1892700c9cb9092e7a"
format_stamp: "Formatted at 2026-09-27 15:07:30 on dist-test-slave-hhpq"
server_key: "3ba40cd01137162db8ab47fbdde4712e"
server_key_iv: "3235eacb60eb17a06aa836caa2164649"
server_key_version: "encryptionkey@0"
I20260927 15:07:30.874441 9642 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:07:30.876973 9657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:30.878355 9642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:30.878481 9642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "1bf59df430f84a1892700c9cb9092e7a"
format_stamp: "Formatted at 2026-09-27 15:07:30 on dist-test-slave-hhpq"
server_key: "3ba40cd01137162db8ab47fbdde4712e"
server_key_iv: "3235eacb60eb17a06aa836caa2164649"
server_key_version: "encryptionkey@0"
I20260927 15:07:30.878590 9642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:30.893378 9642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:30.896729 9642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:30.896952 9642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:30.903899 9642 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:46879
I20260927 15:07:30.903939 9709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:46879 every 8 connection(s)
I20260927 15:07:30.904949 9642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:30.907759 9710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:30.913607 9710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a: Bootstrap starting.
I20260927 15:07:30.914419 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 9642
I20260927 15:07:30.914533 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:07:30.914788 7239 external_mini_cluster.cc:1428] Setting key 118e26fa3b1d3c0792816dd1f7ce5b04
I20260927 15:07:30.916442 9710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:30.917238 9710 log.cc:824] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:30.919157 9710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a: No bootstrap required, opened a new log
I20260927 15:07:30.922607 9710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf59df430f84a1892700c9cb9092e7a" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46879 } }
I20260927 15:07:30.922848 9710 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:30.922904 9710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf59df430f84a1892700c9cb9092e7a, State: Initialized, Role: FOLLOWER
Sep 27 15:07:30 dist-test-slave-hhpq krb5kdc[9629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:30.923360 9710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf59df430f84a1892700c9cb9092e7a" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46879 } }
I20260927 15:07:30.923489 9710 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:30.923550 9710 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:30.923642 9710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:30.924778 9710 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf59df430f84a1892700c9cb9092e7a" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46879 } }
I20260927 15:07:30.925177 9710 leader_election.cc:302] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bf59df430f84a1892700c9cb9092e7a; no voters:
I20260927 15:07:30.925539 9710 leader_election.cc:288] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:30.925630 9715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:30.925879 9715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a [term 1 LEADER]: Becoming Leader. State: Replica: 1bf59df430f84a1892700c9cb9092e7a, State: Running, Role: LEADER
I20260927 15:07:30.926234 9715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf59df430f84a1892700c9cb9092e7a" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46879 } }
I20260927 15:07:30.926888 9710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:30.928063 9717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bf59df430f84a1892700c9cb9092e7a. Latest consensus state: current_term: 1 leader_uuid: "1bf59df430f84a1892700c9cb9092e7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf59df430f84a1892700c9cb9092e7a" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46879 } } }
I20260927 15:07:30.928059 9716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bf59df430f84a1892700c9cb9092e7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf59df430f84a1892700c9cb9092e7a" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46879 } } }
I20260927 15:07:30.928195 9716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:30.928195 9717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:30.929378 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:30.915926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45844 (local address 127.7.17.254:46879)
0927 15:07:30.916349 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:30.916366 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:30.916403 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:07:30.917139 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:30.917160 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:30.917641 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:30.917971 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:30.918763 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:30.919792 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:30.920633 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:30.921170 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:30.921328 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:30.923704 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:30.923711 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:30.923729 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:30.923762 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:30.925625 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:30.926115 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:30.926120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:30.926127 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:30.926263 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:30.926577 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:30.926580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:30.926582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:30.927101 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:30.927302 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:30.927949 (+ 647us) server_negotiation.cc:299] Negotiation successful
0927 15:07:30.928262 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":116,"threads_started":1}
W20260927 15:07:30.932351 9730 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:07:30.932430 9730 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:07:30.932499 9724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:30.933264 9724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:30.939090 9724 catalog_manager.cc:1347] Generated new cluster ID: e9c35a9ab0a542af81bb3cd23082b265
I20260927 15:07:30.939229 9724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:30.947140 9724 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:30.948431 9724 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:30.954619 9724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bf59df430f84a1892700c9cb9092e7a: Generated new TSK 0
I20260927 15:07:30.955549 9724 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
I20260927 15:07:30.985462 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46879
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:31.099140 9736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:31.099439 9736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:31.099483 9736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:31.103590 9736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:31.103674 9736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:31.103744 9736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:07:31.109373 9736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:31.110519 9736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:31.111589 9736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:31.121470 9742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:31.121505 9741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:31.121560 9744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:31.121642 9736 server_base.cc:1034] running on GCE node
I20260927 15:07:31.122131 9736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:31.122803 9736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:31.124011 9736 hybrid_clock.cc:648] HybridClock initialized: now 1790521651123981 us; error 45 us; skew 500 ppm
Sep 27 15:07:31 dist-test-slave-hhpq krb5kdc[9629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521651, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:31.127031 9736 init.cc:377] Logged in from keytab as kudu/127.7.17.193@KRBTEST.COM (short username kudu)
I20260927 15:07:31.128034 9736 webserver.cc:460] Webserver started at http://127.7.17.193:39725/ using document root <none> and password file <none>
I20260927 15:07:31.128611 9736 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:31.128679 9736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:31.128917 9736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:31.131096 9736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "6dbc3eed8b47472ca4d8f4e1ba1a0137"
format_stamp: "Formatted at 2026-09-27 15:07:31 on dist-test-slave-hhpq"
server_key: "38c614b18c19d195ab3a7eb1753b556d"
server_key_iv: "c623c1449300e819c86151898b29bc27"
server_key_version: "encryptionkey@0"
I20260927 15:07:31.131693 9736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "6dbc3eed8b47472ca4d8f4e1ba1a0137"
format_stamp: "Formatted at 2026-09-27 15:07:31 on dist-test-slave-hhpq"
server_key: "38c614b18c19d195ab3a7eb1753b556d"
server_key_iv: "c623c1449300e819c86151898b29bc27"
server_key_version: "encryptionkey@0"
I20260927 15:07:31.135568 9736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 15:07:31.138475 9751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:31.139940 9736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:07:31.140079 9736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "6dbc3eed8b47472ca4d8f4e1ba1a0137"
format_stamp: "Formatted at 2026-09-27 15:07:31 on dist-test-slave-hhpq"
server_key: "38c614b18c19d195ab3a7eb1753b556d"
server_key_iv: "c623c1449300e819c86151898b29bc27"
server_key_version: "encryptionkey@0"
I20260927 15:07:31.140197 9736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:31.176364 9736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:31.180039 9736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:31.180258 9736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:31.180931 9736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:31.181828 9736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:31.181882 9736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:31.181938 9736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:31.181965 9736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:31.191425 9736 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:44455
I20260927 15:07:31.191466 9864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:44455 every 8 connection(s)
I20260927 15:07:31.192556 9736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:31.202590 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 9736
Sep 27 15:07:31 dist-test-slave-hhpq krb5kdc[9629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521651, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:31.202694 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:07:31.202948 7239 external_mini_cluster.cc:1428] Setting key 12ec3e9ba633fbbf8110549b5f117f47
I20260927 15:07:31.207186 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:31.194519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:32987 (local address 127.7.17.254:46879)
0927 15:07:31.194666 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:31.194671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:31.195669 (+ 998us) server_negotiation.cc:407] Connection header received
0927 15:07:31.197054 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:31.197057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:31.197108 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:31.197175 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:31.198617 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:31.199158 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:31.200458 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:31.200693 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:31.200782 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:31.203315 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:31.203325 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:31.203327 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:31.203355 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:31.205049 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:31.205584 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:31.205587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:31.205589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:31.205678 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:31.206062 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:31.206066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:31.206068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:31.206270 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:31.206361 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:31.206889 (+ 528us) server_negotiation.cc:299] Negotiation successful
0927 15:07:31.207022 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:07:31.208091 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:31.194984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46879 (local address 127.7.17.193:32987)
0927 15:07:31.195501 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:31.195552 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:31.196761 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:31.197446 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:31.197454 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:31.197841 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:31.198398 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:31.198414 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:31.199289 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:31.199292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:31.200291 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:31.200303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:31.200856 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:31.200861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:31.201026 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:31.201631 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:31.201648 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:31.203122 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:31.205167 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:31.205173 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:31.205188 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:31.205491 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:31.205763 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:31.205766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:31.205767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:31.205958 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:31.206376 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:31.206384 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:31.206651 (+ 267us) client_negotiation.cc:769] Sending connection context
0927 15:07:31.206832 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 15:07:31.207191 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":129,"threads_started":1}
I20260927 15:07:31.209340 9865 heartbeater.cc:347] Connected to a master server at 127.7.17.254:46879
I20260927 15:07:31.209628 9865 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:31.210268 9865 heartbeater.cc:511] Master 127.7.17.254:46879 requested a full tablet report, sending...
I20260927 15:07:31.211894 9674 ts_manager.cc:194] Registered new tserver with Master: 6dbc3eed8b47472ca4d8f4e1ba1a0137 (127.7.17.193:44455)
WARNING: no policy specified for kudu/127.7.17.194@KRBTEST.COM; defaulting to no policy
I20260927 15:07:31.213109 9674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:32987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
I20260927 15:07:31.229925 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46879
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:31.340764 9870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:31.341090 9870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:31.341133 9870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:31.345297 9870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:31.345368 9870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:31.345430 9870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:07:31.350962 9870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:31.352062 9870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:31.353096 9870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:07:31.362861 9870 server_base.cc:1034] running on GCE node
W20260927 15:07:31.362895 9875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:31.362895 9876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:31.363008 9878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:31.363483 9870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:31.364159 9870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:31.365298 9870 hybrid_clock.cc:648] HybridClock initialized: now 1790521651365261 us; error 50 us; skew 500 ppm
Sep 27 15:07:31 dist-test-slave-hhpq krb5kdc[9629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521651, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:31.368062 9870 init.cc:377] Logged in from keytab as kudu/127.7.17.194@KRBTEST.COM (short username kudu)
I20260927 15:07:31.368996 9870 webserver.cc:460] Webserver started at http://127.7.17.194:39459/ using document root <none> and password file <none>
I20260927 15:07:31.369557 9870 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:31.369647 9870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:31.369868 9870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:31.371950 9870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "3437d1d4470e4f59a590292e8396feb1"
format_stamp: "Formatted at 2026-09-27 15:07:31 on dist-test-slave-hhpq"
server_key: "5701e4e9d48b79ee1c5a99a7e8f8e83c"
server_key_iv: "74b4ae9a423be9229c3f66b6efcab904"
server_key_version: "encryptionkey@0"
I20260927 15:07:31.372488 9870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "3437d1d4470e4f59a590292e8396feb1"
format_stamp: "Formatted at 2026-09-27 15:07:31 on dist-test-slave-hhpq"
server_key: "5701e4e9d48b79ee1c5a99a7e8f8e83c"
server_key_iv: "74b4ae9a423be9229c3f66b6efcab904"
server_key_version: "encryptionkey@0"
I20260927 15:07:31.376389 9870 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:07:31.379051 9885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:31.380393 9870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 15:07:31.380523 9870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "3437d1d4470e4f59a590292e8396feb1"
format_stamp: "Formatted at 2026-09-27 15:07:31 on dist-test-slave-hhpq"
server_key: "5701e4e9d48b79ee1c5a99a7e8f8e83c"
server_key_iv: "74b4ae9a423be9229c3f66b6efcab904"
server_key_version: "encryptionkey@0"
I20260927 15:07:31.380645 9870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:31.405536 9870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:31.408598 9870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:31.408780 9870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:31.409497 9870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:31.410349 9870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:31.410398 9870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:31.410439 9870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:31.410468 9870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:31.419018 9870 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:35913
I20260927 15:07:31.419049 9998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:35913 every 8 connection(s)
I20260927 15:07:31.420007 9870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:07:31.426467 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 9870
I20260927 15:07:31.426573 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:07:31.426818 7239 external_mini_cluster.cc:1428] Setting key 7d2bcec3fea153c43670b38dc2d2c216
Sep 27 15:07:31 dist-test-slave-hhpq krb5kdc[9629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521651, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:31.435029 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:31.422014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:59521 (local address 127.7.17.254:46879)
0927 15:07:31.422183 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:31.422187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:31.423094 (+ 907us) server_negotiation.cc:407] Connection header received
0927 15:07:31.424550 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:31.424554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:31.424614 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:31.424689 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:31.425953 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:31.426487 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:31.427764 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:31.428082 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:31.428187 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:31.431049 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:31.431053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:31.431056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:31.431079 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:31.432605 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:31.433211 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:31.433214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:31.433216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:31.433460 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:31.433864 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:31.433866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:31.433868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:31.434067 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:31.434164 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:31.434725 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 15:07:31.434861 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
WARNING: no policy specified for kudu/127.7.17.195@KRBTEST.COM; defaulting to no policy
I20260927 15:07:31.435917 10001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:31.422412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46879 (local address 127.7.17.194:59521)
0927 15:07:31.422923 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:31.422973 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:31.424323 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:31.424930 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:31.424938 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:31.425315 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:31.425813 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:31.425828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:31.426613 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:31.426617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:31.427655 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:31.427664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:31.428234 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:31.428238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:31.428439 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:31.429114 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:31.429130 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:31.430866 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:31.432735 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:31.432742 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:31.432761 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:31.433100 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:31.433586 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:31.433590 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:31.433592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:31.433760 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:31.434185 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:31.434193 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:31.434476 (+ 283us) client_negotiation.cc:769] Sending connection context
0927 15:07:31.434663 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 15:07:31.435029 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":122,"threads_started":1}
I20260927 15:07:31.437322 9999 heartbeater.cc:347] Connected to a master server at 127.7.17.254:46879
I20260927 15:07:31.437646 9999 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.195@KRBTEST.COM" created.
I20260927 15:07:31.438275 9999 heartbeater.cc:511] Master 127.7.17.254:46879 requested a full tablet report, sending...
I20260927 15:07:31.439584 9674 ts_manager.cc:194] Registered new tserver with Master: 3437d1d4470e4f59a590292e8396feb1 (127.7.17.194:35913)
I20260927 15:07:31.440174 9674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:59521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
I20260927 15:07:31.454278 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46879
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:31.565893 10004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:31.566195 10004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:31.566259 10004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:31.570468 10004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:31.570544 10004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:31.570628 10004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:07:31.576233 10004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:31.577446 10004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:31.578469 10004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:31.588467 10009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:31.588608 10004 server_base.cc:1034] running on GCE node
W20260927 15:07:31.588441 10010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:31.588788 10012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:31.589267 10004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:31.589922 10004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:31.591104 10004 hybrid_clock.cc:648] HybridClock initialized: now 1790521651591070 us; error 35 us; skew 500 ppm
Sep 27 15:07:31 dist-test-slave-hhpq krb5kdc[9629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521651, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:31.593986 10004 init.cc:377] Logged in from keytab as kudu/127.7.17.195@KRBTEST.COM (short username kudu)
I20260927 15:07:31.594957 10004 webserver.cc:460] Webserver started at http://127.7.17.195:44375/ using document root <none> and password file <none>
I20260927 15:07:31.595516 10004 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:31.595584 10004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:31.595790 10004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:31.597976 10004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "dd6efbf8e58f40a79920fcc78aaf472b"
format_stamp: "Formatted at 2026-09-27 15:07:31 on dist-test-slave-hhpq"
server_key: "cc711ba4c85013b701ca3997eb22c192"
server_key_iv: "e45edfb376bb1d285951c3c53f7774c8"
server_key_version: "encryptionkey@0"
I20260927 15:07:31.598556 10004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "dd6efbf8e58f40a79920fcc78aaf472b"
format_stamp: "Formatted at 2026-09-27 15:07:31 on dist-test-slave-hhpq"
server_key: "cc711ba4c85013b701ca3997eb22c192"
server_key_iv: "e45edfb376bb1d285951c3c53f7774c8"
server_key_version: "encryptionkey@0"
I20260927 15:07:31.602363 10004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:07:31.604923 10019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:31.606252 10004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:07:31.606383 10004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "dd6efbf8e58f40a79920fcc78aaf472b"
format_stamp: "Formatted at 2026-09-27 15:07:31 on dist-test-slave-hhpq"
server_key: "cc711ba4c85013b701ca3997eb22c192"
server_key_iv: "e45edfb376bb1d285951c3c53f7774c8"
server_key_version: "encryptionkey@0"
I20260927 15:07:31.606508 10004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:31.620649 10004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:31.623857 10004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:31.624037 10004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:31.624625 10004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:31.625478 10004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:31.625525 10004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:31.625578 10004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:31.625605 10004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:31.634457 10004 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:36697
I20260927 15:07:31.634485 10132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:36697 every 8 connection(s)
I20260927 15:07:31.635370 10004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:31.640507 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 10004
I20260927 15:07:31.640641 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:07:31.640959 7239 external_mini_cluster.cc:1428] Setting key e65b318ee27a399d2be013bdc108ebb8
Sep 27 15:07:31 dist-test-slave-hhpq krb5kdc[9629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521651, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:31.649552 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:31.637157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:53783 (local address 127.7.17.254:46879)
0927 15:07:31.637287 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:31.637292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:31.638279 (+ 987us) server_negotiation.cc:407] Connection header received
0927 15:07:31.639579 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:31.639583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:31.639642 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:31.639716 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:31.641287 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:31.641949 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:31.643507 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:31.643721 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:31.643796 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:31.646312 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:31.646315 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:31.646317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:31.646342 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:31.647644 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:31.648174 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:31.648177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:31.648178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:31.648261 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:31.648548 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:31.648550 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:31.648551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:31.648720 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:31.648796 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:31.649325 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 15:07:31.649425 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:07:31.650403 10135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:31.637583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46879 (local address 127.7.17.195:53783)
0927 15:07:31.638058 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:31.638106 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:31.639347 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:31.639917 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:31.639924 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:31.640346 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:31.641098 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:31.641121 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:31.642119 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:31.642123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:31.643392 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:31.643402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:31.643865 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:31.643868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:31.644018 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:31.644618 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:31.644642 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:31.646132 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:31.647781 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:31.647786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:31.647806 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:31.648069 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:31.648339 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:31.648341 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:31.648342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:31.648477 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:31.648800 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:31.648806 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:31.649093 (+ 287us) client_negotiation.cc:769] Sending connection context
0927 15:07:31.649271 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 15:07:31.649538 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":123,"threads_started":1}
I20260927 15:07:31.651578 10133 heartbeater.cc:347] Connected to a master server at 127.7.17.254:46879
I20260927 15:07:31.651850 10133 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:31.652395 10133 heartbeater.cc:511] Master 127.7.17.254:46879 requested a full tablet report, sending...
I20260927 15:07:31.653471 9674 ts_manager.cc:194] Registered new tserver with Master: dd6efbf8e58f40a79920fcc78aaf472b (127.7.17.195:36697)
I20260927 15:07:31.654047 9674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:53783
I20260927 15:07:31.655700 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 15:07:31.668361 9713 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:31.666808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45850 (local address 127.7.17.254:46879)
0927 15:07:31.666980 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:31.666984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:31.667079 (+ 95us) server_negotiation.cc:407] Connection header received
0927 15:07:31.667196 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:31.667198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:31.667248 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:31.667320 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:31.668157 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45850: BlockingRecv error: recv got EOF from 127.0.0.1:45850 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 15:07:31.669661 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 9736
I20260927 15:07:31.674578 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 9870
I20260927 15:07:31.679402 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 10004
I20260927 15:07:31.684422 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 9642
2026-09-27T15:07:31Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2034 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:07:31 dist-test-slave-hhpq krb5kdc[10145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:07:31 dist-test-slave-hhpq krb5kdc[10145](info): set up 2 sockets
Sep 27 15:07:31 dist-test-slave-hhpq krb5kdc[10145](info): commencing operation
krb5kdc: starting...
W20260927 15:07:32.644675 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:07:32 dist-test-slave-hhpq krb5kdc[10145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:07:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:07:32.766089 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:41387
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:41387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:32.877744 10158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:32.878048 10158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:32.878108 10158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:32.882354 10158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:32.882418 10158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:32.882452 10158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:32.882475 10158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:32.882504 10158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:32.888064 10158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:41387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:41387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:32.889278 10158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:32.890331 10158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:32.899251 10166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:32.899278 10163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:32.899325 10164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:32.899482 10158 server_base.cc:1034] running on GCE node
I20260927 15:07:32.900238 10158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:32.901551 10158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:32.902760 10158 hybrid_clock.cc:648] HybridClock initialized: now 1790521652902721 us; error 36 us; skew 500 ppm
Sep 27 15:07:32 dist-test-slave-hhpq krb5kdc[10145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521652, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:32.909948 10158 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:07:32.911026 10158 webserver.cc:460] Webserver started at http://127.7.17.254:38753/ using document root <none> and password file <none>
I20260927 15:07:32.911607 10158 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:32.911676 10158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:32.911919 10158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:32.914196 10158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "59f9c4e856d8403f8d52c844d686dc30"
format_stamp: "Formatted at 2026-09-27 15:07:32 on dist-test-slave-hhpq"
server_key: "933d425ad211dd257848a2e986c92a47"
server_key_iv: "bc34b1dcf6e0e20034dae2d55b49035d"
server_key_version: "encryptionkey@0"
I20260927 15:07:32.914773 10158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "59f9c4e856d8403f8d52c844d686dc30"
format_stamp: "Formatted at 2026-09-27 15:07:32 on dist-test-slave-hhpq"
server_key: "933d425ad211dd257848a2e986c92a47"
server_key_iv: "bc34b1dcf6e0e20034dae2d55b49035d"
server_key_version: "encryptionkey@0"
I20260927 15:07:32.918793 10158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:07:32.921594 10173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:32.923118 10158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:32.923251 10158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "59f9c4e856d8403f8d52c844d686dc30"
format_stamp: "Formatted at 2026-09-27 15:07:32 on dist-test-slave-hhpq"
server_key: "933d425ad211dd257848a2e986c92a47"
server_key_iv: "bc34b1dcf6e0e20034dae2d55b49035d"
server_key_version: "encryptionkey@0"
I20260927 15:07:32.923367 10158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:32.950520 10158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:32.953909 10158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:32.954099 10158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:32.960759 10158 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:41387
I20260927 15:07:32.961026 10225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:41387 every 8 connection(s)
I20260927 15:07:32.961736 10158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:32.962767 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 10158
I20260927 15:07:32.962858 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:07:32.963117 7239 external_mini_cluster.cc:1428] Setting key b9176870f83bf70f526288c3ace3006d
I20260927 15:07:32.964732 10226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:32.970901 10226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30: Bootstrap starting.
Sep 27 15:07:32 dist-test-slave-hhpq krb5kdc[10145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:32.973359 10226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:32.974047 10226 log.cc:824] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:32.976145 10226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30: No bootstrap required, opened a new log
I20260927 15:07:32.976992 10229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:32.964585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38240 (local address 127.7.17.254:41387)
0927 15:07:32.964995 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:32.965012 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:32.965046 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:07:32.965657 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:32.965672 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:32.965996 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:32.966266 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:32.967158 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:32.968138 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:32.969310 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:32.969793 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:32.969915 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:32.972610 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:32.972614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:32.972627 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:32.972655 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:32.974299 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:32.974715 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:32.974721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:32.974727 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:32.974841 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:32.975104 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:32.975106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:32.975107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:32.975490 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:32.975633 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:32.975899 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 15:07:32.976167 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":113,"threads_started":1}
I20260927 15:07:32.980113 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:07:32.980137 10226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f9c4e856d8403f8d52c844d686dc30" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41387 } }
I20260927 15:07:32.980408 10226 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:32.980468 10226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f9c4e856d8403f8d52c844d686dc30, State: Initialized, Role: FOLLOWER
I20260927 15:07:32.980971 10226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f9c4e856d8403f8d52c844d686dc30" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41387 } }
I20260927 15:07:32.981128 10226 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:32.981187 10226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:32.981282 10226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:32.982427 10226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f9c4e856d8403f8d52c844d686dc30" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41387 } }
I20260927 15:07:32.982787 10226 leader_election.cc:302] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59f9c4e856d8403f8d52c844d686dc30; no voters:
I20260927 15:07:32.983091 10226 leader_election.cc:288] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:32.983191 10232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:32.983431 10232 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30 [term 1 LEADER]: Becoming Leader. State: Replica: 59f9c4e856d8403f8d52c844d686dc30, State: Running, Role: LEADER
I20260927 15:07:32.983902 10232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f9c4e856d8403f8d52c844d686dc30" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41387 } }
I20260927 15:07:32.984635 10226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:32.985375 10232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59f9c4e856d8403f8d52c844d686dc30. Latest consensus state: current_term: 1 leader_uuid: "59f9c4e856d8403f8d52c844d686dc30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f9c4e856d8403f8d52c844d686dc30" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41387 } } }
I20260927 15:07:32.985505 10232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:32.985754 10234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59f9c4e856d8403f8d52c844d686dc30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f9c4e856d8403f8d52c844d686dc30" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41387 } } }
I20260927 15:07:32.985869 10234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30 [sys.catalog]: This master's current role is: LEADER
W20260927 15:07:32.989949 10248 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:07:32.990049 10248 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:07:32.990118 10237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:32.990820 10237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:32.997220 10237 catalog_manager.cc:1347] Generated new cluster ID: af89c64c34cd4d4d8cdfa33878bb2d36
I20260927 15:07:32.997295 10237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:32.998080 10229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:32.992857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.7.17.254:41387)
0927 15:07:32.992977 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:32.992981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:32.993278 (+ 297us) server_negotiation.cc:407] Connection header received
0927 15:07:32.993414 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:32.993417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:32.993470 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:32.993559 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:32.994203 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:32.994913 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:32.995867 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:32.996120 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:32.996209 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:32.997064 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:32.997067 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:32.997070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:32.997601 (+ 531us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:32.997637 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:32.997657 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:32.997745 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:32.997885 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 15:07:32.997948 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:07:33.008705 10237 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:33.009827 10237 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:33.016999 10237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59f9c4e856d8403f8d52c844d686dc30: Generated new TSK 0
I20260927 15:07:33.017918 10237 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:07:33.024859 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 10158
2026-09-27T15:07:33Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1338 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:07:33 dist-test-slave-hhpq krb5kdc[10259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:07:33 dist-test-slave-hhpq krb5kdc[10259](info): set up 2 sockets
Sep 27 15:07:33 dist-test-slave-hhpq krb5kdc[10259](info): commencing operation
krb5kdc: starting...
W20260927 15:07:33.972637 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:07:34 dist-test-slave-hhpq krb5kdc[10259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:07:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:07:34.091405 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:39071
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:37607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:39071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:34.198465 10272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:34.198750 10272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:34.198791 10272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:34.203683 10272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:34.203760 10272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:34.203780 10272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:34.203796 10272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:34.203810 10272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:34.209268 10272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:37607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:39071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:39071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:34.210304 10272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:34.211287 10272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:34.219491 10280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:34.219511 10277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:34.219514 10278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:34.219560 10272 server_base.cc:1034] running on GCE node
I20260927 15:07:34.220332 10272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:34.221465 10272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:34.222618 10272 hybrid_clock.cc:648] HybridClock initialized: now 1790521654222607 us; error 27 us; skew 500 ppm
Sep 27 15:07:34 dist-test-slave-hhpq krb5kdc[10259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521654, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:34.225438 10272 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:07:34.226341 10272 webserver.cc:460] Webserver started at http://127.7.17.254:46445/ using document root <none> and password file <none>
I20260927 15:07:34.226898 10272 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:34.226961 10272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:34.227151 10272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:34.229333 10272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "1ed101efb69c4358943b5570d82f44a8"
format_stamp: "Formatted at 2026-09-27 15:07:34 on dist-test-slave-hhpq"
server_key: "1f16c51f554e9e7162fcd0919e2ea804"
server_key_iv: "2bc92078bebbaafc2e23e6aed16708f7"
server_key_version: "encryptionkey@0"
I20260927 15:07:34.229899 10272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "1ed101efb69c4358943b5570d82f44a8"
format_stamp: "Formatted at 2026-09-27 15:07:34 on dist-test-slave-hhpq"
server_key: "1f16c51f554e9e7162fcd0919e2ea804"
server_key_iv: "2bc92078bebbaafc2e23e6aed16708f7"
server_key_version: "encryptionkey@0"
I20260927 15:07:34.233664 10272 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:07:34.236048 10287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:34.237393 10272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:07:34.237497 10272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "1ed101efb69c4358943b5570d82f44a8"
format_stamp: "Formatted at 2026-09-27 15:07:34 on dist-test-slave-hhpq"
server_key: "1f16c51f554e9e7162fcd0919e2ea804"
server_key_iv: "2bc92078bebbaafc2e23e6aed16708f7"
server_key_version: "encryptionkey@0"
I20260927 15:07:34.237599 10272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:34.254150 10272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:34.257596 10272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:34.257791 10272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:34.264739 10272 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:39071
I20260927 15:07:34.265051 10339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:39071 every 8 connection(s)
I20260927 15:07:34.265800 10272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:34.267256 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 10272
I20260927 15:07:34.267367 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:07:34.267666 7239 external_mini_cluster.cc:1428] Setting key 353cef357f64b45b48d6fabbb404822e
I20260927 15:07:34.269045 10340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:34.274663 10340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed101efb69c4358943b5570d82f44a8: Bootstrap starting.
Sep 27 15:07:34 dist-test-slave-hhpq krb5kdc[10259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:34.277062 10340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ed101efb69c4358943b5570d82f44a8: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:34.277899 10340 log.cc:824] T 00000000000000000000000000000000 P 1ed101efb69c4358943b5570d82f44a8: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:34.280198 10340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed101efb69c4358943b5570d82f44a8: No bootstrap required, opened a new log
I20260927 15:07:34.281723 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:34.269159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38184 (local address 127.7.17.254:39071)
0927 15:07:34.269595 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:34.269609 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:34.269645 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:07:34.270293 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:34.270310 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:34.270649 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:34.270975 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:34.271763 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:34.272798 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:34.273650 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:34.274134 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:34.274280 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:34.276859 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:34.276868 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:34.276887 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:34.276917 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:34.278769 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:34.279252 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:34.279257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:34.279262 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:34.279375 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:34.279707 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:34.279710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:34.279711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:34.280146 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:34.280331 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:34.280597 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 15:07:34.280947 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":138,"threads_started":1}
I20260927 15:07:34.283910 10340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ed101efb69c4358943b5570d82f44a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed101efb69c4358943b5570d82f44a8" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39071 } }
I20260927 15:07:34.284124 10340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ed101efb69c4358943b5570d82f44a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:34.284170 10340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ed101efb69c4358943b5570d82f44a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed101efb69c4358943b5570d82f44a8, State: Initialized, Role: FOLLOWER
I20260927 15:07:34.284614 10340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ed101efb69c4358943b5570d82f44a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed101efb69c4358943b5570d82f44a8" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39071 } }
I20260927 15:07:34.284742 10340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ed101efb69c4358943b5570d82f44a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:34.284802 10340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ed101efb69c4358943b5570d82f44a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:34.284912 10340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ed101efb69c4358943b5570d82f44a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:34.285782 10340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ed101efb69c4358943b5570d82f44a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed101efb69c4358943b5570d82f44a8" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39071 } }
I20260927 15:07:34.286089 10340 leader_election.cc:302] T 00000000000000000000000000000000 P 1ed101efb69c4358943b5570d82f44a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ed101efb69c4358943b5570d82f44a8; no voters:
I20260927 15:07:34.286379 10340 leader_election.cc:288] T 00000000000000000000000000000000 P 1ed101efb69c4358943b5570d82f44a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:34.286514 10345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ed101efb69c4358943b5570d82f44a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:34.286727 10345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ed101efb69c4358943b5570d82f44a8 [term 1 LEADER]: Becoming Leader. State: Replica: 1ed101efb69c4358943b5570d82f44a8, State: Running, Role: LEADER
I20260927 15:07:34.287012 10345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ed101efb69c4358943b5570d82f44a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed101efb69c4358943b5570d82f44a8" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39071 } }
I20260927 15:07:34.287604 10340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ed101efb69c4358943b5570d82f44a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:34.288792 10346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed101efb69c4358943b5570d82f44a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ed101efb69c4358943b5570d82f44a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed101efb69c4358943b5570d82f44a8" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39071 } } }
I20260927 15:07:34.288959 10346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed101efb69c4358943b5570d82f44a8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:34.288789 10347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed101efb69c4358943b5570d82f44a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ed101efb69c4358943b5570d82f44a8. Latest consensus state: current_term: 1 leader_uuid: "1ed101efb69c4358943b5570d82f44a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed101efb69c4358943b5570d82f44a8" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39071 } } }
I20260927 15:07:34.289093 10347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed101efb69c4358943b5570d82f44a8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:34.289371 10355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:34.292018 10355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:34.297848 10355 catalog_manager.cc:1347] Generated new cluster ID: 8cb5713554504b29b9f231b0b2c68d99
I20260927 15:07:34.297936 10355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:34.310513 10355 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:34.311815 10355 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:34.322124 10355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ed101efb69c4358943b5570d82f44a8: Generated new TSK 0
I20260927 15:07:34.323077 10355 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
I20260927 15:07:34.356110 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39071
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:37607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:34.471946 10366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:34.472223 10366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:34.472267 10366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:34.476315 10366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:34.476388 10366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:34.476451 10366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:07:34.481796 10366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:37607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:34.482980 10366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:34.484021 10366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:34.493762 10371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:34.493770 10374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:34.493796 10372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:34.494163 10366 server_base.cc:1034] running on GCE node
I20260927 15:07:34.494590 10366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:34.495250 10366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:34.496438 10366 hybrid_clock.cc:648] HybridClock initialized: now 1790521654496414 us; error 35 us; skew 500 ppm
Sep 27 15:07:34 dist-test-slave-hhpq krb5kdc[10259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521654, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:34.499325 10366 init.cc:377] Logged in from keytab as kudu/127.7.17.193@KRBTEST.COM (short username kudu)
I20260927 15:07:34.500319 10366 webserver.cc:460] Webserver started at http://127.7.17.193:40967/ using document root <none> and password file <none>
I20260927 15:07:34.500921 10366 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:34.500991 10366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:34.501189 10366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:34.503271 10366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "ceb8c312338e43c6839efe6de8a3ea78"
format_stamp: "Formatted at 2026-09-27 15:07:34 on dist-test-slave-hhpq"
server_key: "98e358339d1c14c0dcce3a6e877f2bd2"
server_key_iv: "fd2f8c13db05967a0dcd96812df48f9c"
server_key_version: "encryptionkey@0"
I20260927 15:07:34.503866 10366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "ceb8c312338e43c6839efe6de8a3ea78"
format_stamp: "Formatted at 2026-09-27 15:07:34 on dist-test-slave-hhpq"
server_key: "98e358339d1c14c0dcce3a6e877f2bd2"
server_key_iv: "fd2f8c13db05967a0dcd96812df48f9c"
server_key_version: "encryptionkey@0"
I20260927 15:07:34.507817 10366 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 15:07:34.510547 10381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:34.511900 10366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:07:34.512017 10366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "ceb8c312338e43c6839efe6de8a3ea78"
format_stamp: "Formatted at 2026-09-27 15:07:34 on dist-test-slave-hhpq"
server_key: "98e358339d1c14c0dcce3a6e877f2bd2"
server_key_iv: "fd2f8c13db05967a0dcd96812df48f9c"
server_key_version: "encryptionkey@0"
I20260927 15:07:34.512122 10366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:34.545866 10366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:34.549062 10366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:34.549266 10366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:34.549855 10366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:34.550674 10366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:34.550731 10366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:34.550786 10366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:34.550809 10366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:34.559311 10366 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:42979
I20260927 15:07:34.559343 10494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:42979 every 8 connection(s)
I20260927 15:07:34.560184 10366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:34.562731 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 10366
I20260927 15:07:34.562844 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:07:34.563114 7239 external_mini_cluster.cc:1428] Setting key b2c97219b7363eeaf6e41044ad5501f8
Sep 27 15:07:34 dist-test-slave-hhpq krb5kdc[10259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521654, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.194@KRBTEST.COM" created.
I20260927 15:07:34.575938 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:34.562047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:50813 (local address 127.7.17.254:39071)
0927 15:07:34.562191 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:34.562194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:34.563123 (+ 929us) server_negotiation.cc:407] Connection header received
0927 15:07:34.564592 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:34.564595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:34.564649 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:34.564748 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:34.566159 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:34.567016 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:34.568728 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:34.569148 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:34.569282 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:34.572309 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:34.572313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:34.572316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:34.572352 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:34.573742 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:34.574246 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:34.574249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:34.574250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:34.574326 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:34.574640 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:34.574643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:34.574645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:34.574866 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:34.574975 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:34.575625 (+ 650us) server_negotiation.cc:299] Negotiation successful
0927 15:07:34.575766 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:07:34.576995 10497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:34.562440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:39071 (local address 127.7.17.193:50813)
0927 15:07:34.562948 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:34.563003 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:34.564355 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:34.564944 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:34.564952 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:34.565395 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:34.565977 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:34.566001 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:34.567164 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:34.567167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:34.568574 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:34.568587 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:34.569350 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:34.569355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:34.569576 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:34.570285 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:34.570310 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:34.572124 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:34.573856 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:34.573862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:34.573876 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:34.574165 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:34.574397 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:34.574399 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:34.574400 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:34.574548 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:34.575005 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:34.575012 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:34.575352 (+ 340us) client_negotiation.cc:769] Sending connection context
0927 15:07:34.575549 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 15:07:34.575839 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":117,"threads_started":1}
I20260927 15:07:34.578423 10495 heartbeater.cc:347] Connected to a master server at 127.7.17.254:39071
I20260927 15:07:34.578711 10495 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:34.579355 10495 heartbeater.cc:511] Master 127.7.17.254:39071 requested a full tablet report, sending...
I20260927 15:07:34.581123 10304 ts_manager.cc:194] Registered new tserver with Master: ceb8c312338e43c6839efe6de8a3ea78 (127.7.17.193:42979)
I20260927 15:07:34.582315 10304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:50813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
I20260927 15:07:34.591451 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39071
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:37607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:34.701983 10500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:34.702301 10500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:34.702374 10500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:34.706504 10500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:34.706581 10500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:34.706661 10500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:07:34.712155 10500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:37607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:34.713517 10500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:34.714744 10500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:34.724726 10505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:34.724777 10506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:34.724881 10500 server_base.cc:1034] running on GCE node
W20260927 15:07:34.724772 10508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:34.725452 10500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:34.726136 10500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:34.727332 10500 hybrid_clock.cc:648] HybridClock initialized: now 1790521654727313 us; error 32 us; skew 500 ppm
Sep 27 15:07:34 dist-test-slave-hhpq krb5kdc[10259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521654, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:34.730298 10500 init.cc:377] Logged in from keytab as kudu/127.7.17.194@KRBTEST.COM (short username kudu)
I20260927 15:07:34.731245 10500 webserver.cc:460] Webserver started at http://127.7.17.194:39889/ using document root <none> and password file <none>
I20260927 15:07:34.731799 10500 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:34.731866 10500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:34.732081 10500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:34.734205 10500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "93f0b9df8df245708a6ba7f5c3b3dddb"
format_stamp: "Formatted at 2026-09-27 15:07:34 on dist-test-slave-hhpq"
server_key: "c32088c103a1588061934f5d19c87b47"
server_key_iv: "bd84601c0c9f2f667441984e6e3a9fc0"
server_key_version: "encryptionkey@0"
I20260927 15:07:34.734766 10500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "93f0b9df8df245708a6ba7f5c3b3dddb"
format_stamp: "Formatted at 2026-09-27 15:07:34 on dist-test-slave-hhpq"
server_key: "c32088c103a1588061934f5d19c87b47"
server_key_iv: "bd84601c0c9f2f667441984e6e3a9fc0"
server_key_version: "encryptionkey@0"
I20260927 15:07:34.738971 10500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:07:34.741755 10515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:34.743171 10500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:34.743355 10500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "93f0b9df8df245708a6ba7f5c3b3dddb"
format_stamp: "Formatted at 2026-09-27 15:07:34 on dist-test-slave-hhpq"
server_key: "c32088c103a1588061934f5d19c87b47"
server_key_iv: "bd84601c0c9f2f667441984e6e3a9fc0"
server_key_version: "encryptionkey@0"
I20260927 15:07:34.743468 10500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:34.775671 10500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:34.779142 10500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:34.779404 10500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:34.780032 10500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:34.780936 10500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:34.780984 10500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:34.781038 10500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:34.781066 10500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:34.789827 10500 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:40487
I20260927 15:07:34.789870 10628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:40487 every 8 connection(s)
I20260927 15:07:34.790803 10500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:07:34.798152 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 10500
I20260927 15:07:34.798260 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:07:34.798555 7239 external_mini_cluster.cc:1428] Setting key e90aa2eb298b72aa4bb9657733e2516d
Sep 27 15:07:34 dist-test-slave-hhpq krb5kdc[10259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521654, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:34.805646 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:34.792751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:45283 (local address 127.7.17.254:39071)
0927 15:07:34.792989 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:34.792993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:34.793862 (+ 869us) server_negotiation.cc:407] Connection header received
0927 15:07:34.795290 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:34.795294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:34.795357 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:34.795432 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:34.796721 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:34.797313 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:34.798582 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:34.798850 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:34.798954 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:34.801993 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:34.801997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:34.802000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:34.802027 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:34.803496 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:34.804049 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:34.804052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:34.804054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:34.804132 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:34.804463 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:34.804467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:34.804468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:34.804665 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:34.804753 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:34.805351 (+ 598us) server_negotiation.cc:299] Negotiation successful
0927 15:07:34.805483 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260927 15:07:34.806547 10631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:34.793189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:39071 (local address 127.7.17.194:45283)
0927 15:07:34.793671 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:34.793740 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:34.795042 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:34.795629 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:34.795637 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:34.796037 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:34.796558 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:34.796573 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:34.797449 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:34.797451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:34.798452 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:34.798463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:34.799015 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:34.799018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:34.799195 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:34.799862 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:34.799879 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:34.801781 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:34.803618 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:34.803624 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:34.803644 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:34.803948 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:34.804215 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:34.804218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:34.804220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:34.804372 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:34.804765 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:34.804773 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:34.805099 (+ 326us) client_negotiation.cc:769] Sending connection context
0927 15:07:34.805280 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 15:07:34.805634 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.7.17.195@KRBTEST.COM; defaulting to no policy
I20260927 15:07:34.807974 10629 heartbeater.cc:347] Connected to a master server at 127.7.17.254:39071
I20260927 15:07:34.808234 10629 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:34.808768 10629 heartbeater.cc:511] Master 127.7.17.254:39071 requested a full tablet report, sending...
I20260927 15:07:34.809890 10304 ts_manager.cc:194] Registered new tserver with Master: 93f0b9df8df245708a6ba7f5c3b3dddb (127.7.17.194:40487)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.195@KRBTEST.COM" created.
I20260927 15:07:34.810678 10304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:45283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
I20260927 15:07:34.826011 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39071
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:37607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:34.934998 10634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:34.935294 10634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:34.935355 10634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:34.939486 10634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:34.939567 10634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:34.939649 10634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:07:34.945261 10634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:37607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:34.946353 10634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:34.947471 10634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:07:34.957113 10634 server_base.cc:1034] running on GCE node
W20260927 15:07:34.957093 10639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:34.957291 10642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:34.957073 10640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:34.957715 10634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:34.958355 10634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:34.959523 10634 hybrid_clock.cc:648] HybridClock initialized: now 1790521654959489 us; error 48 us; skew 500 ppm
Sep 27 15:07:34 dist-test-slave-hhpq krb5kdc[10259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521654, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:34.962198 10634 init.cc:377] Logged in from keytab as kudu/127.7.17.195@KRBTEST.COM (short username kudu)
I20260927 15:07:34.963094 10634 webserver.cc:460] Webserver started at http://127.7.17.195:42351/ using document root <none> and password file <none>
I20260927 15:07:34.963621 10634 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:34.963680 10634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:34.963860 10634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:34.965981 10634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "6a9d7375ce7a4368a5e1588bcc062c4b"
format_stamp: "Formatted at 2026-09-27 15:07:34 on dist-test-slave-hhpq"
server_key: "438c7c591261c33f40aaac3aa7f4d0cd"
server_key_iv: "a4e2dc69f9cac3ef8bff70b1bc659987"
server_key_version: "encryptionkey@0"
I20260927 15:07:34.966537 10634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "6a9d7375ce7a4368a5e1588bcc062c4b"
format_stamp: "Formatted at 2026-09-27 15:07:34 on dist-test-slave-hhpq"
server_key: "438c7c591261c33f40aaac3aa7f4d0cd"
server_key_iv: "a4e2dc69f9cac3ef8bff70b1bc659987"
server_key_version: "encryptionkey@0"
I20260927 15:07:34.970410 10634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:07:34.972941 10649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:34.974213 10634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 15:07:34.974318 10634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "6a9d7375ce7a4368a5e1588bcc062c4b"
format_stamp: "Formatted at 2026-09-27 15:07:34 on dist-test-slave-hhpq"
server_key: "438c7c591261c33f40aaac3aa7f4d0cd"
server_key_iv: "a4e2dc69f9cac3ef8bff70b1bc659987"
server_key_version: "encryptionkey@0"
I20260927 15:07:34.974421 10634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:34.994000 10634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:34.996935 10634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:34.997107 10634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:34.997696 10634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:34.998507 10634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:34.998550 10634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:34.998590 10634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:34.998618 10634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:35.006955 10634 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:35509
I20260927 15:07:35.006981 10762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:35509 every 8 connection(s)
I20260927 15:07:35.007829 10634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:35.012410 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 10634
I20260927 15:07:35.012529 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:07:35.012866 7239 external_mini_cluster.cc:1428] Setting key 69a65673384be9156a8086108ddefae7
Sep 27 15:07:35 dist-test-slave-hhpq krb5kdc[10259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521654, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:35.021867 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:35.009693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:34897 (local address 127.7.17.254:39071)
0927 15:07:35.009858 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:35.009863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:35.010699 (+ 836us) server_negotiation.cc:407] Connection header received
0927 15:07:35.012067 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:35.012070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:35.012117 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:35.012197 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:35.013738 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:35.014345 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:35.015668 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:35.015889 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:35.015966 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:35.018470 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:35.018473 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:35.018475 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:35.018501 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:35.019806 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:35.020327 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:35.020330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:35.020332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:35.020412 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:35.020696 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:35.020698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:35.020700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:35.020936 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:35.021013 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:35.021611 (+ 598us) server_negotiation.cc:299] Negotiation successful
0927 15:07:35.021731 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:07:35.022715 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:35.010057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:39071 (local address 127.7.17.195:34897)
0927 15:07:35.010539 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:35.010585 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:35.011829 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:35.012435 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:35.012446 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:35.012991 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:35.013592 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:35.013610 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:35.014471 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:35.014474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:35.015559 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:35.015570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:35.016001 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:35.016004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:35.016143 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:35.016719 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:35.016741 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:35.018294 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:35.019920 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:35.019925 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:35.019940 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:35.020199 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:35.020483 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:35.020485 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:35.020487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:35.020622 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:35.021047 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:35.021053 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:35.021348 (+ 295us) client_negotiation.cc:769] Sending connection context
0927 15:07:35.021523 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 15:07:35.021818 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":137,"threads_started":1}
I20260927 15:07:35.023926 10763 heartbeater.cc:347] Connected to a master server at 127.7.17.254:39071
I20260927 15:07:35.024183 10763 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:35.024703 10763 heartbeater.cc:511] Master 127.7.17.254:39071 requested a full tablet report, sending...
I20260927 15:07:35.025909 10304 ts_manager.cc:194] Registered new tserver with Master: 6a9d7375ce7a4368a5e1588bcc062c4b (127.7.17.195:35509)
I20260927 15:07:35.026571 10304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:34897
I20260927 15:07:35.026930 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:07:35 dist-test-slave-hhpq krb5kdc[10259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521655, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 15:07:35 dist-test-slave-hhpq krb5kdc[10259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521655, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:35.052242 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:35.043487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45082 (local address 127.7.17.254:39071)
0927 15:07:35.043645 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:35.043650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:35.043815 (+ 165us) server_negotiation.cc:407] Connection header received
0927 15:07:35.043949 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:35.043953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:35.043997 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:35.044066 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:35.044865 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:35.045403 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:35.046231 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:35.046452 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:35.046555 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:35.049104 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:35.049108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:35.049110 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:35.049138 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:35.050497 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:35.050947 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:35.050950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:35.050951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:35.051040 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:35.051362 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:35.051365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:35.051366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:35.051591 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:35.051710 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:35.051954 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 15:07:35.052070 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 15:07:35.052726 10304 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45082
I20260927 15:07:35.053819 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 10366
I20260927 15:07:35.058607 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 10500
I20260927 15:07:35.063349 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 10634
I20260927 15:07:35.067736 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 10272
2026-09-27T15:07:35Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2044 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:07:35 dist-test-slave-hhpq krb5kdc[10775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:07:35 dist-test-slave-hhpq krb5kdc[10775](info): set up 2 sockets
Sep 27 15:07:35 dist-test-slave-hhpq krb5kdc[10775](info): commencing operation
krb5kdc: starting...
W20260927 15:07:36.028710 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:07:36 dist-test-slave-hhpq krb5kdc[10775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:07:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:07:36.148806 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:41171
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:34575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:41171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:36.258778 10788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:36.259069 10788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:36.259110 10788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:36.263134 10788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:36.263216 10788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:36.263234 10788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:36.263250 10788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:36.263264 10788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:36.268672 10788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:34575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:41171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:41171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:36.269788 10788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:36.270758 10788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:07:36.279496 10788 server_base.cc:1034] running on GCE node
W20260927 15:07:36.279445 10793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:36.279445 10796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:36.279444 10794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:36.280304 10788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:36.281477 10788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:36.282660 10788 hybrid_clock.cc:648] HybridClock initialized: now 1790521656282633 us; error 44 us; skew 500 ppm
Sep 27 15:07:36 dist-test-slave-hhpq krb5kdc[10775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521656, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:36.285521 10788 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:07:36.286459 10788 webserver.cc:460] Webserver started at http://127.7.17.254:38301/ using document root <none> and password file <none>
I20260927 15:07:36.286995 10788 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:36.287055 10788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:36.287235 10788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:36.289371 10788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "099ed2afaa17436892b1bea263cecd72"
format_stamp: "Formatted at 2026-09-27 15:07:36 on dist-test-slave-hhpq"
server_key: "45eb7886f628b81d01c9e90f56892e65"
server_key_iv: "3428e2c277e80ed9e092d6e046d26a4c"
server_key_version: "encryptionkey@0"
I20260927 15:07:36.289918 10788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "099ed2afaa17436892b1bea263cecd72"
format_stamp: "Formatted at 2026-09-27 15:07:36 on dist-test-slave-hhpq"
server_key: "45eb7886f628b81d01c9e90f56892e65"
server_key_iv: "3428e2c277e80ed9e092d6e046d26a4c"
server_key_version: "encryptionkey@0"
I20260927 15:07:36.293777 10788 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:07:36.296348 10803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:36.297825 10788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:07:36.297959 10788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "099ed2afaa17436892b1bea263cecd72"
format_stamp: "Formatted at 2026-09-27 15:07:36 on dist-test-slave-hhpq"
server_key: "45eb7886f628b81d01c9e90f56892e65"
server_key_iv: "3428e2c277e80ed9e092d6e046d26a4c"
server_key_version: "encryptionkey@0"
I20260927 15:07:36.298060 10788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:36.321985 10788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:36.325311 10788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:36.325470 10788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:36.332238 10788 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:41171
I20260927 15:07:36.332357 10855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:41171 every 8 connection(s)
I20260927 15:07:36.333240 10788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:36.335103 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 10788
I20260927 15:07:36.335276 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:07:36.335579 7239 external_mini_cluster.cc:1428] Setting key 6fc152acdc0292372be3c3257ca3044f
I20260927 15:07:36.336347 10856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:36.342707 10856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 099ed2afaa17436892b1bea263cecd72: Bootstrap starting.
Sep 27 15:07:36 dist-test-slave-hhpq krb5kdc[10775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:36.345443 10856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 099ed2afaa17436892b1bea263cecd72: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:36.346226 10856 log.cc:824] T 00000000000000000000000000000000 P 099ed2afaa17436892b1bea263cecd72: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:36.348512 10856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 099ed2afaa17436892b1bea263cecd72: No bootstrap required, opened a new log
I20260927 15:07:36.350332 10859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:36.337010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52388 (local address 127.7.17.254:41171)
0927 15:07:36.337414 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:36.337430 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:36.337466 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:07:36.338142 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:36.338160 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:36.338495 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:36.338788 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:36.339699 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:36.340706 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:36.341917 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:36.342397 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:36.342519 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:36.345281 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:36.345286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:36.345300 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:36.345329 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:36.347299 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:36.347860 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:36.347866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:36.347870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:36.347995 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:36.348311 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:36.348313 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:36.348314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:36.348777 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:36.348986 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:36.349270 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 15:07:36.349594 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":123,"threads_started":1}
I20260927 15:07:36.352272 10856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 099ed2afaa17436892b1bea263cecd72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "099ed2afaa17436892b1bea263cecd72" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41171 } }
I20260927 15:07:36.352497 10856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 099ed2afaa17436892b1bea263cecd72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:36.352541 10856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 099ed2afaa17436892b1bea263cecd72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 099ed2afaa17436892b1bea263cecd72, State: Initialized, Role: FOLLOWER
I20260927 15:07:36.352984 10856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 099ed2afaa17436892b1bea263cecd72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "099ed2afaa17436892b1bea263cecd72" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41171 } }
I20260927 15:07:36.353112 10856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 099ed2afaa17436892b1bea263cecd72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:36.353174 10856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 099ed2afaa17436892b1bea263cecd72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:36.353264 10856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 099ed2afaa17436892b1bea263cecd72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:36.354171 10856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 099ed2afaa17436892b1bea263cecd72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "099ed2afaa17436892b1bea263cecd72" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41171 } }
I20260927 15:07:36.354498 10856 leader_election.cc:302] T 00000000000000000000000000000000 P 099ed2afaa17436892b1bea263cecd72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 099ed2afaa17436892b1bea263cecd72; no voters:
I20260927 15:07:36.354790 10856 leader_election.cc:288] T 00000000000000000000000000000000 P 099ed2afaa17436892b1bea263cecd72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:36.354915 10861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 099ed2afaa17436892b1bea263cecd72 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:36.355124 10861 raft_consensus.cc:686] T 00000000000000000000000000000000 P 099ed2afaa17436892b1bea263cecd72 [term 1 LEADER]: Becoming Leader. State: Replica: 099ed2afaa17436892b1bea263cecd72, State: Running, Role: LEADER
I20260927 15:07:36.355419 10861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 099ed2afaa17436892b1bea263cecd72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "099ed2afaa17436892b1bea263cecd72" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41171 } }
I20260927 15:07:36.355919 10856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 099ed2afaa17436892b1bea263cecd72 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:36.357103 10862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 099ed2afaa17436892b1bea263cecd72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "099ed2afaa17436892b1bea263cecd72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "099ed2afaa17436892b1bea263cecd72" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41171 } } }
I20260927 15:07:36.357208 10862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 099ed2afaa17436892b1bea263cecd72 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:36.357481 10863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 099ed2afaa17436892b1bea263cecd72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 099ed2afaa17436892b1bea263cecd72. Latest consensus state: current_term: 1 leader_uuid: "099ed2afaa17436892b1bea263cecd72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "099ed2afaa17436892b1bea263cecd72" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41171 } } }
I20260927 15:07:36.357566 10863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 099ed2afaa17436892b1bea263cecd72 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:36.357653 10870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:36.360320 10870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:36.366004 10870 catalog_manager.cc:1347] Generated new cluster ID: 0f43d3397f434561a0c8ce75829082d3
I20260927 15:07:36.366086 10870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:36.377381 10870 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:36.378360 10870 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:36.385213 10870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 099ed2afaa17436892b1bea263cecd72: Generated new TSK 0
I20260927 15:07:36.385937 10870 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
I20260927 15:07:36.419806 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:41171
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:34575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:36.530052 10882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:36.530339 10882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:36.530382 10882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:36.534497 10882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:36.534574 10882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:36.534639 10882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:07:36.540218 10882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:34575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:41171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:36.541353 10882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:36.542402 10882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:36.552006 10890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:36.552006 10888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:36.552104 10882 server_base.cc:1034] running on GCE node
W20260927 15:07:36.552065 10887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:36.552719 10882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:36.553565 10882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:36.554741 10882 hybrid_clock.cc:648] HybridClock initialized: now 1790521656554706 us; error 48 us; skew 500 ppm
Sep 27 15:07:36 dist-test-slave-hhpq krb5kdc[10775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521656, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:36.557830 10882 init.cc:377] Logged in from keytab as kudu/127.7.17.193@KRBTEST.COM (short username kudu)
I20260927 15:07:36.558800 10882 webserver.cc:460] Webserver started at http://127.7.17.193:37633/ using document root <none> and password file <none>
I20260927 15:07:36.559363 10882 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:36.559432 10882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:36.559633 10882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:36.561807 10882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "90bbdc4bf1bd498c85559b9d7e8dde7e"
format_stamp: "Formatted at 2026-09-27 15:07:36 on dist-test-slave-hhpq"
server_key: "c93cf84951d1c3fc0cc120fd78c8257c"
server_key_iv: "a40dd58c1ebfff409074ee48750cdf33"
server_key_version: "encryptionkey@0"
I20260927 15:07:36.562368 10882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "90bbdc4bf1bd498c85559b9d7e8dde7e"
format_stamp: "Formatted at 2026-09-27 15:07:36 on dist-test-slave-hhpq"
server_key: "c93cf84951d1c3fc0cc120fd78c8257c"
server_key_iv: "a40dd58c1ebfff409074ee48750cdf33"
server_key_version: "encryptionkey@0"
I20260927 15:07:36.566203 10882 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:07:36.568979 10897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:36.570348 10882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:36.570492 10882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "90bbdc4bf1bd498c85559b9d7e8dde7e"
format_stamp: "Formatted at 2026-09-27 15:07:36 on dist-test-slave-hhpq"
server_key: "c93cf84951d1c3fc0cc120fd78c8257c"
server_key_iv: "a40dd58c1ebfff409074ee48750cdf33"
server_key_version: "encryptionkey@0"
I20260927 15:07:36.570613 10882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:36.593825 10882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:36.596971 10882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:36.597141 10882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:36.597723 10882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:36.598534 10882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:36.598577 10882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:36.598618 10882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:36.598646 10882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:36.606989 10882 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:38331
I20260927 15:07:36.607010 11010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:38331 every 8 connection(s)
I20260927 15:07:36.607864 10882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:36.616156 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 10882
I20260927 15:07:36.616351 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:07:36.616609 7239 external_mini_cluster.cc:1428] Setting key e316d2637bfbe9d626eb0ad752e20f56
Sep 27 15:07:36 dist-test-slave-hhpq krb5kdc[10775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521656, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:36.622056 10859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:36.609754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:38061 (local address 127.7.17.254:41171)
0927 15:07:36.609895 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:36.609900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:36.610767 (+ 867us) server_negotiation.cc:407] Connection header received
0927 15:07:36.612094 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:36.612097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:36.612144 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:36.612228 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:36.613464 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:36.614016 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:36.615160 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:36.615408 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:36.615493 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:36.618367 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:36.618370 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:36.618372 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:36.618396 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:36.619813 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:36.620422 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:36.620425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:36.620426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:36.620504 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:36.620878 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:36.620882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:36.620884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:36.621069 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:36.621145 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:36.621805 (+ 660us) server_negotiation.cc:299] Negotiation successful
0927 15:07:36.621934 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:07:36.623325 11013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:36.610161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:41171 (local address 127.7.17.193:38061)
0927 15:07:36.610600 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:36.610642 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:36.611880 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:36.612384 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:36.612390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:36.612761 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:36.613300 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:36.613314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:36.614139 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:36.614142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:36.615033 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:36.615041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:36.615573 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:36.615577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:36.615750 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:36.616450 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:36.616472 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:36.618202 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:36.619926 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:36.619933 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:36.619951 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:36.620317 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:36.620580 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:36.620583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:36.620585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:36.620761 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:36.621163 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:36.621170 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:36.621511 (+ 341us) client_negotiation.cc:769] Sending connection context
0927 15:07:36.621736 (+ 225us) client_negotiation.cc:240] Negotiation successful
0927 15:07:36.622099 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":101,"threads_started":1}
I20260927 15:07:36.624727 11011 heartbeater.cc:347] Connected to a master server at 127.7.17.254:41171
I20260927 15:07:36.625005 11011 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:36.625552 11011 heartbeater.cc:511] Master 127.7.17.254:41171 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.17.194@KRBTEST.COM; defaulting to no policy
I20260927 15:07:36.627112 10820 ts_manager.cc:194] Registered new tserver with Master: 90bbdc4bf1bd498c85559b9d7e8dde7e (127.7.17.193:38331)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.194@KRBTEST.COM" created.
I20260927 15:07:36.628412 10820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:38061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
I20260927 15:07:36.643301 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:41171
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:34575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:36.755283 11016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:36.755568 11016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:36.755626 11016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:36.759858 11016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:36.759927 11016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:36.760010 11016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:07:36.765730 11016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:34575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:41171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:36.766908 11016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:36.767999 11016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:36.777948 11021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:36.778030 11016 server_base.cc:1034] running on GCE node
W20260927 15:07:36.777971 11022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:36.777974 11024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:36.778576 11016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:36.779273 11016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:36.780439 11016 hybrid_clock.cc:648] HybridClock initialized: now 1790521656780428 us; error 28 us; skew 500 ppm
Sep 27 15:07:36 dist-test-slave-hhpq krb5kdc[10775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521656, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:36.783351 11016 init.cc:377] Logged in from keytab as kudu/127.7.17.194@KRBTEST.COM (short username kudu)
I20260927 15:07:36.784291 11016 webserver.cc:460] Webserver started at http://127.7.17.194:46005/ using document root <none> and password file <none>
I20260927 15:07:36.784868 11016 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:36.784934 11016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:36.785125 11016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:36.787271 11016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "54ba6fd8f2f14cc1a34032cdade8837b"
format_stamp: "Formatted at 2026-09-27 15:07:36 on dist-test-slave-hhpq"
server_key: "e7a60d70419d57fc7aacf19ff9131a64"
server_key_iv: "8918b1b6008d3f9cd91139f879040e65"
server_key_version: "encryptionkey@0"
I20260927 15:07:36.787832 11016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "54ba6fd8f2f14cc1a34032cdade8837b"
format_stamp: "Formatted at 2026-09-27 15:07:36 on dist-test-slave-hhpq"
server_key: "e7a60d70419d57fc7aacf19ff9131a64"
server_key_iv: "8918b1b6008d3f9cd91139f879040e65"
server_key_version: "encryptionkey@0"
I20260927 15:07:36.791735 11016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:07:36.794409 11031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:36.795759 11016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:07:36.795884 11016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "54ba6fd8f2f14cc1a34032cdade8837b"
format_stamp: "Formatted at 2026-09-27 15:07:36 on dist-test-slave-hhpq"
server_key: "e7a60d70419d57fc7aacf19ff9131a64"
server_key_iv: "8918b1b6008d3f9cd91139f879040e65"
server_key_version: "encryptionkey@0"
I20260927 15:07:36.795989 11016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:36.809664 11016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:36.812649 11016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:36.812851 11016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:36.813484 11016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:36.814289 11016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:36.814335 11016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:36.814388 11016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:36.814417 11016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:36.822793 11016 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:42693
I20260927 15:07:36.822857 11144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:42693 every 8 connection(s)
I20260927 15:07:36.823705 11016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:07:36.829691 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 11016
I20260927 15:07:36.829809 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:07:36.830128 7239 external_mini_cluster.cc:1428] Setting key cd8c275a6bb77dd65086dbb5d339304e
Sep 27 15:07:36 dist-test-slave-hhpq krb5kdc[10775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521656, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:36.838621 10859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:36.825569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:41177 (local address 127.7.17.254:41171)
0927 15:07:36.825689 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:36.825693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:36.826537 (+ 844us) server_negotiation.cc:407] Connection header received
0927 15:07:36.828022 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:36.828025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:36.828083 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:36.828157 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:36.829499 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:36.830176 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:36.831621 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:36.831867 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:36.831945 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:36.834738 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:36.834741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:36.834743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:36.834768 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:36.836324 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:36.836983 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:36.836987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:36.836989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:36.837098 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:36.837504 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:36.837506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:36.837507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:36.837683 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:36.837754 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:36.838397 (+ 643us) server_negotiation.cc:299] Negotiation successful
0927 15:07:36.838500 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for kudu/127.7.17.195@KRBTEST.COM; defaulting to no policy
I20260927 15:07:36.839854 11147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:36.825930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:41171 (local address 127.7.17.194:41177)
0927 15:07:36.826380 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:36.826424 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:36.827795 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:36.828367 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:36.828375 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:36.828765 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:36.829343 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:36.829357 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:36.830317 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:36.830321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:36.831498 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:36.831507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:36.832027 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:36.832031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:36.832237 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:36.832864 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:36.832886 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:36.834556 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:36.836456 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:36.836463 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:36.836498 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:36.836882 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:36.837205 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:36.837208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:36.837210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:36.837410 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:36.837775 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:36.837782 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:36.838129 (+ 347us) client_negotiation.cc:769] Sending connection context
0927 15:07:36.838360 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 15:07:36.838721 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":103,"threads_started":1}
I20260927 15:07:36.841061 11145 heartbeater.cc:347] Connected to a master server at 127.7.17.254:41171
I20260927 15:07:36.841301 11145 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:36.841809 11145 heartbeater.cc:511] Master 127.7.17.254:41171 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.195@KRBTEST.COM" created.
I20260927 15:07:36.842833 10820 ts_manager.cc:194] Registered new tserver with Master: 54ba6fd8f2f14cc1a34032cdade8837b (127.7.17.194:42693)
I20260927 15:07:36.843444 10820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:41177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
I20260927 15:07:36.858335 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:41171
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:34575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:36.969712 11150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:36.969992 11150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:36.970034 11150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:36.974269 11150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:36.974327 11150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:36.974395 11150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:07:36.979827 11150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:34575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:41171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:36.980959 11150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:36.982002 11150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:07:36.991797 11150 server_base.cc:1034] running on GCE node
W20260927 15:07:36.991760 11155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:36.991763 11156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:36.991760 11158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:36.992430 11150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:36.993098 11150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:36.994294 11150 hybrid_clock.cc:648] HybridClock initialized: now 1790521656994275 us; error 32 us; skew 500 ppm
Sep 27 15:07:36 dist-test-slave-hhpq krb5kdc[10775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521656, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:36.997103 11150 init.cc:377] Logged in from keytab as kudu/127.7.17.195@KRBTEST.COM (short username kudu)
I20260927 15:07:36.998072 11150 webserver.cc:460] Webserver started at http://127.7.17.195:39075/ using document root <none> and password file <none>
I20260927 15:07:36.998636 11150 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:36.998714 11150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:36.998915 11150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:37.001101 11150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "db145864d0e34a3b9f9ff0e1c47a8175"
format_stamp: "Formatted at 2026-09-27 15:07:36 on dist-test-slave-hhpq"
server_key: "79fd130729cb90990bbb4424eb7d8859"
server_key_iv: "ab81e1d568ba4991a8a87159d46fab1b"
server_key_version: "encryptionkey@0"
I20260927 15:07:37.001657 11150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "db145864d0e34a3b9f9ff0e1c47a8175"
format_stamp: "Formatted at 2026-09-27 15:07:36 on dist-test-slave-hhpq"
server_key: "79fd130729cb90990bbb4424eb7d8859"
server_key_iv: "ab81e1d568ba4991a8a87159d46fab1b"
server_key_version: "encryptionkey@0"
I20260927 15:07:37.005723 11150 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:07:37.008239 11165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:37.009620 11150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:37.009735 11150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "db145864d0e34a3b9f9ff0e1c47a8175"
format_stamp: "Formatted at 2026-09-27 15:07:36 on dist-test-slave-hhpq"
server_key: "79fd130729cb90990bbb4424eb7d8859"
server_key_iv: "ab81e1d568ba4991a8a87159d46fab1b"
server_key_version: "encryptionkey@0"
I20260927 15:07:37.009843 11150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:37.021108 11150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:37.024411 11150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:37.024598 11150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:37.025271 11150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:37.026117 11150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:37.026165 11150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:37.026229 11150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:37.026258 11150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:37.034691 11150 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:43305
I20260927 15:07:37.034718 11278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:43305 every 8 connection(s)
I20260927 15:07:37.035607 11150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:37.044857 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 11150
Sep 27 15:07:37 dist-test-slave-hhpq krb5kdc[10775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521656, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:37.044996 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:07:37.045295 7239 external_mini_cluster.cc:1428] Setting key 53d7392d03e1bab321916e0ec157a273
I20260927 15:07:37.049088 10859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:37.037438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:50005 (local address 127.7.17.254:41171)
0927 15:07:37.037566 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:37.037570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:37.038394 (+ 824us) server_negotiation.cc:407] Connection header received
0927 15:07:37.039766 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:37.039769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:37.039826 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:37.039901 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:37.041162 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:37.041685 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:37.042830 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:37.043044 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:37.043135 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:37.045569 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:37.045572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:37.045574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:37.045601 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:37.047097 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:37.047632 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:37.047636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:37.047638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:37.047716 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:37.048024 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:37.048027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:37.048028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:37.048205 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:37.048294 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:37.048781 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 15:07:37.048930 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:07:37.049971 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:37.037805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:41171 (local address 127.7.17.195:50005)
0927 15:07:37.038234 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:37.038288 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:37.039545 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:37.040082 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:37.040102 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:37.040482 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:37.041017 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:37.041033 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:37.041808 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:37.041811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:37.042702 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:37.042711 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:37.043188 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:37.043192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:37.043353 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:37.043911 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:37.043928 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:37.045401 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:37.047204 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:37.047210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:37.047227 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:37.047546 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:37.047793 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:37.047797 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:37.047799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:37.047941 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:37.048303 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:37.048310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:37.048560 (+ 250us) client_negotiation.cc:769] Sending connection context
0927 15:07:37.048727 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 15:07:37.049093 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":99,"threads_started":1}
I20260927 15:07:37.051151 11279 heartbeater.cc:347] Connected to a master server at 127.7.17.254:41171
I20260927 15:07:37.051412 11279 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:37.051947 11279 heartbeater.cc:511] Master 127.7.17.254:41171 requested a full tablet report, sending...
I20260927 15:07:37.053104 10820 ts_manager.cc:194] Registered new tserver with Master: db145864d0e34a3b9f9ff0e1c47a8175 (127.7.17.195:43305)
I20260927 15:07:37.053753 10820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:50005
I20260927 15:07:37.059554 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:07:37 dist-test-slave-hhpq krb5kdc[10775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:07:37 dist-test-slave-hhpq krb5kdc[10775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.193@KRBTEST.COM
I20260927 15:07:37.084770 11286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:37.075451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57722 (local address 127.7.17.193:38331)
0927 15:07:37.075743 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:37.075757 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:37.075786 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:07:37.075835 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:37.075840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:37.076025 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:37.076147 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:37.076979 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:37.077807 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:37.078705 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:37.079094 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:37.079179 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:37.081570 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:37.081579 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:37.081594 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:37.081621 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:37.083268 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:37.083618 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:37.083623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:37.083624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:37.083702 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:37.083882 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:37.083884 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:37.083885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:37.084116 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:37.084332 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:37.084442 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 15:07:37.084570 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":106,"threads_started":1}
Sep 27 15:07:37 dist-test-slave-hhpq krb5kdc[10775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:37.095789 10859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:37.087681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52394 (local address 127.7.17.254:41171)
0927 15:07:37.087848 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:37.087853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:37.087929 (+ 76us) server_negotiation.cc:407] Connection header received
0927 15:07:37.088045 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:37.088048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:37.088093 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:37.088162 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:37.088754 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:37.089481 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:37.090257 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:37.090483 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:37.090567 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:37.092927 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:37.092931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:37.092934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:37.092964 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:37.094239 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:37.094607 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:37.094611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:37.094613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:37.094688 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:37.094924 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:37.094927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:37.094929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:37.095148 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:37.095323 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:37.095524 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 15:07:37.095632 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 15:07:37.096186 10820 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52394
I20260927 15:07:37.096866 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 10882
I20260927 15:07:37.102116 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 11016
I20260927 15:07:37.106628 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 11150
I20260927 15:07:37.111235 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 10788
2026-09-27T15:07:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2043 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:07:37 dist-test-slave-hhpq krb5kdc[11291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:07:37 dist-test-slave-hhpq krb5kdc[11291](info): set up 2 sockets
Sep 27 15:07:37 dist-test-slave-hhpq krb5kdc[11291](info): commencing operation
krb5kdc: starting...
W20260927 15:07:38.068729 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:07:38 dist-test-slave-hhpq krb5kdc[11291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:07:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:07:38.192137 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:46873
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:46873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:38.304239 11304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:38.304536 11304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:38.304577 11304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:38.308732 11304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:38.308808 11304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:38.308848 11304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:38.308866 11304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:38.308878 11304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:38.314683 11304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:46873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:46873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:38.315775 11304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:38.316759 11304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:38.325129 11309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:38.325163 11304 server_base.cc:1034] running on GCE node
W20260927 15:07:38.325169 11312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:38.325440 11310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:38.326033 11304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:38.327183 11304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:38.328341 11304 hybrid_clock.cc:648] HybridClock initialized: now 1790521658328311 us; error 47 us; skew 500 ppm
Sep 27 15:07:38 dist-test-slave-hhpq krb5kdc[11291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521658, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:38.336373 11304 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:07:38.337225 11304 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:38.337294 11304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:38.337493 11304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:38.339705 11304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "5999c65ba7ab498a9cb7c8f8cbd7266c"
format_stamp: "Formatted at 2026-09-27 15:07:38 on dist-test-slave-hhpq"
server_key: "8c81d4cb5e165d7962ac7764ecfbf0a2"
server_key_iv: "e9693b1ea7550320047bd4d4d33c8b53"
server_key_version: "encryptionkey@0"
I20260927 15:07:38.340269 11304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "5999c65ba7ab498a9cb7c8f8cbd7266c"
format_stamp: "Formatted at 2026-09-27 15:07:38 on dist-test-slave-hhpq"
server_key: "8c81d4cb5e165d7962ac7764ecfbf0a2"
server_key_iv: "e9693b1ea7550320047bd4d4d33c8b53"
server_key_version: "encryptionkey@0"
I20260927 15:07:38.344164 11304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:07:38.346868 11318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:38.348229 11304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:07:38.348361 11304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "5999c65ba7ab498a9cb7c8f8cbd7266c"
format_stamp: "Formatted at 2026-09-27 15:07:38 on dist-test-slave-hhpq"
server_key: "8c81d4cb5e165d7962ac7764ecfbf0a2"
server_key_iv: "e9693b1ea7550320047bd4d4d33c8b53"
server_key_version: "encryptionkey@0"
I20260927 15:07:38.348469 11304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:38.363333 11304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:38.366691 11304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:38.366884 11304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:38.374024 11304 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:46873
I20260927 15:07:38.374409 11370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:46873 every 8 connection(s)
I20260927 15:07:38.374744 11304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:38.377705 11371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:38.378520 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 11304
I20260927 15:07:38.378659 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:07:38.378914 7239 external_mini_cluster.cc:1428] Setting key a6abfee1743c775348865d4ec6d1da88
I20260927 15:07:38.384560 11371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c: Bootstrap starting.
I20260927 15:07:38.387259 11371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:38.388016 11371 log.cc:824] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:07:38 dist-test-slave-hhpq krb5kdc[11291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:38.390172 11371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c: No bootstrap required, opened a new log
I20260927 15:07:38.393711 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:38.380327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44952 (local address 127.7.17.254:46873)
0927 15:07:38.380770 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:38.380790 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:38.380866 (+ 76us) server_negotiation.cc:407] Connection header received
0927 15:07:38.381680 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:38.381706 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:38.382147 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:38.382555 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:38.383271 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:38.384706 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:38.385651 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:38.386363 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:38.386520 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:38.389171 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:38.389178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:38.389199 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:38.389232 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:38.390833 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:38.391302 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:38.391307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:38.391312 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:38.391423 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:38.391726 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:38.391729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:38.391730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:38.392146 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:38.392299 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:38.392597 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 15:07:38.392910 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":113,"threads_started":1}
I20260927 15:07:38.394261 11371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5999c65ba7ab498a9cb7c8f8cbd7266c" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46873 } }
I20260927 15:07:38.394507 11371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:38.394564 11371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5999c65ba7ab498a9cb7c8f8cbd7266c, State: Initialized, Role: FOLLOWER
I20260927 15:07:38.395061 11371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5999c65ba7ab498a9cb7c8f8cbd7266c" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46873 } }
I20260927 15:07:38.395229 11371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:38.395292 11371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:38.395380 11371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:38.396476 11371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5999c65ba7ab498a9cb7c8f8cbd7266c" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46873 } }
I20260927 15:07:38.396790 11371 leader_election.cc:302] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5999c65ba7ab498a9cb7c8f8cbd7266c; no voters:
I20260927 15:07:38.397101 11371 leader_election.cc:288] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:38.397344 11376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:38.397536 11376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c [term 1 LEADER]: Becoming Leader. State: Replica: 5999c65ba7ab498a9cb7c8f8cbd7266c, State: Running, Role: LEADER
I20260927 15:07:38.397843 11376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5999c65ba7ab498a9cb7c8f8cbd7266c" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46873 } }
I20260927 15:07:38.398387 11371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:38.399652 11377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5999c65ba7ab498a9cb7c8f8cbd7266c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5999c65ba7ab498a9cb7c8f8cbd7266c" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46873 } } }
I20260927 15:07:38.399782 11377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:38.400028 11378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5999c65ba7ab498a9cb7c8f8cbd7266c. Latest consensus state: current_term: 1 leader_uuid: "5999c65ba7ab498a9cb7c8f8cbd7266c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5999c65ba7ab498a9cb7c8f8cbd7266c" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46873 } } }
I20260927 15:07:38.400120 11378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:38.400257 11385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:38.403168 11385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:38.409385 11385 catalog_manager.cc:1347] Generated new cluster ID: 11cc15806bb7443eb2afe9e7a484d626
I20260927 15:07:38.409507 11385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:38.427100 11385 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:38.428525 11385 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:38.435921 11385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c: Generated new TSK 0
I20260927 15:07:38.436808 11385 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
I20260927 15:07:38.472432 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46873
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:38.587496 11397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:38.587800 11397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:38.587848 11397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:38.591977 11397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:38.592057 11397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:38.592121 11397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:07:38.597539 11397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:38.598690 11397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:38.599819 11397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:38.609664 11405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:38.609757 11397 server_base.cc:1034] running on GCE node
W20260927 15:07:38.609615 11402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:38.609912 11403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:38.610316 11397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:38.610980 11397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:38.612129 11397 hybrid_clock.cc:648] HybridClock initialized: now 1790521658612082 us; error 61 us; skew 500 ppm
Sep 27 15:07:38 dist-test-slave-hhpq krb5kdc[11291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521658, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:38.614943 11397 init.cc:377] Logged in from keytab as kudu/127.7.17.193@KRBTEST.COM (short username kudu)
I20260927 15:07:38.615650 11397 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:38.615712 11397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:38.615895 11397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:38.618088 11397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "2a96bccb9df940199cbc9870e732e96a"
format_stamp: "Formatted at 2026-09-27 15:07:38 on dist-test-slave-hhpq"
server_key: "ecac3d21938cb33ed1d79b2c2026ae35"
server_key_iv: "19557b07bf3dda948746e900476d5488"
server_key_version: "encryptionkey@0"
I20260927 15:07:38.618793 11397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "2a96bccb9df940199cbc9870e732e96a"
format_stamp: "Formatted at 2026-09-27 15:07:38 on dist-test-slave-hhpq"
server_key: "ecac3d21938cb33ed1d79b2c2026ae35"
server_key_iv: "19557b07bf3dda948746e900476d5488"
server_key_version: "encryptionkey@0"
I20260927 15:07:38.622641 11397 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:07:38.625213 11411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:38.626519 11397 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:07:38.626623 11397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "2a96bccb9df940199cbc9870e732e96a"
format_stamp: "Formatted at 2026-09-27 15:07:38 on dist-test-slave-hhpq"
server_key: "ecac3d21938cb33ed1d79b2c2026ae35"
server_key_iv: "19557b07bf3dda948746e900476d5488"
server_key_version: "encryptionkey@0"
I20260927 15:07:38.626720 11397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:38.641402 11397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:38.644747 11397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:38.644979 11397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:38.645603 11397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:38.646541 11397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:38.646591 11397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:38.646631 11397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:38.646660 11397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:38.655429 11397 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:44699
I20260927 15:07:38.655449 11524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:44699 every 8 connection(s)
I20260927 15:07:38.656038 11397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:38.658915 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 11397
I20260927 15:07:38.659024 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:07:38.659294 7239 external_mini_cluster.cc:1428] Setting key c686170bb9a69914fbfdb1060a0c841f
Sep 27 15:07:38 dist-test-slave-hhpq krb5kdc[11291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521658, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.17.194@KRBTEST.COM; defaulting to no policy
I20260927 15:07:38.670006 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:38.657749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:34917 (local address 127.7.17.254:46873)
0927 15:07:38.657870 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:38.657874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:38.658730 (+ 856us) server_negotiation.cc:407] Connection header received
0927 15:07:38.660118 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:38.660121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:38.660178 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:38.660245 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:38.661567 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:38.662093 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:38.663247 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:38.663559 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:38.663672 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:38.666190 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:38.666194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:38.666197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:38.666229 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:38.667779 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:38.668333 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:38.668337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:38.668339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:38.668434 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:38.668814 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:38.668816 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:38.668817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:38.669054 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:38.669153 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:38.669761 (+ 608us) server_negotiation.cc:299] Negotiation successful
0927 15:07:38.669878 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:07:38.670832 11527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:38.658106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46873 (local address 127.7.17.193:34917)
0927 15:07:38.658577 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:38.658624 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:38.659907 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:38.660427 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:38.660437 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:38.660909 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:38.661422 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:38.661436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:38.662220 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:38.662223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:38.663111 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:38.663130 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:38.663719 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:38.663722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:38.663889 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:38.664487 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:38.664504 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:38.666006 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:38.667901 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:38.667907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:38.667921 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:38.668232 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:38.668534 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:38.668536 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:38.668537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:38.668680 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:38.669213 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:38.669219 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:38.669488 (+ 269us) client_negotiation.cc:769] Sending connection context
0927 15:07:38.669682 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 15:07:38.669957 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.194@KRBTEST.COM" created.
I20260927 15:07:38.672019 11525 heartbeater.cc:347] Connected to a master server at 127.7.17.254:46873
I20260927 15:07:38.672254 11525 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:38.672729 11525 heartbeater.cc:511] Master 127.7.17.254:46873 requested a full tablet report, sending...
I20260927 15:07:38.674373 11334 ts_manager.cc:194] Registered new tserver with Master: 2a96bccb9df940199cbc9870e732e96a (127.7.17.193:44699)
I20260927 15:07:38.675956 11334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:34917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
I20260927 15:07:38.688476 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46873
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:38.801609 11530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:38.801903 11530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:38.801970 11530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:38.806226 11530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:38.806291 11530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:38.806383 11530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:07:38.811911 11530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:38.813077 11530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:38.814137 11530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:38.823609 11538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:38.823773 11536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:38.823609 11535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:38.823865 11530 server_base.cc:1034] running on GCE node
I20260927 15:07:38.824445 11530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:38.825265 11530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:38.826481 11530 hybrid_clock.cc:648] HybridClock initialized: now 1790521658826451 us; error 49 us; skew 500 ppm
Sep 27 15:07:38 dist-test-slave-hhpq krb5kdc[11291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521658, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:38.829418 11530 init.cc:377] Logged in from keytab as kudu/127.7.17.194@KRBTEST.COM (short username kudu)
I20260927 15:07:38.830214 11530 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:38.830281 11530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:38.830479 11530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:38.832638 11530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "273358c3d1594ae49cb107dedeeda600"
format_stamp: "Formatted at 2026-09-27 15:07:38 on dist-test-slave-hhpq"
server_key: "c549d520c316b74038bcc05a869c36d2"
server_key_iv: "70bb5af8e438c3447991bd983775e90c"
server_key_version: "encryptionkey@0"
I20260927 15:07:38.833233 11530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "273358c3d1594ae49cb107dedeeda600"
format_stamp: "Formatted at 2026-09-27 15:07:38 on dist-test-slave-hhpq"
server_key: "c549d520c316b74038bcc05a869c36d2"
server_key_iv: "70bb5af8e438c3447991bd983775e90c"
server_key_version: "encryptionkey@0"
I20260927 15:07:38.837044 11530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:07:38.839507 11544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:38.840787 11530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:07:38.840960 11530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "273358c3d1594ae49cb107dedeeda600"
format_stamp: "Formatted at 2026-09-27 15:07:38 on dist-test-slave-hhpq"
server_key: "c549d520c316b74038bcc05a869c36d2"
server_key_iv: "70bb5af8e438c3447991bd983775e90c"
server_key_version: "encryptionkey@0"
I20260927 15:07:38.841082 11530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:38.861089 11530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:38.864282 11530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:38.864471 11530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:38.865144 11530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:38.866014 11530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:38.866061 11530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:38.866106 11530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:38.866133 11530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:38.874778 11530 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:45751
I20260927 15:07:38.874822 11657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:45751 every 8 connection(s)
I20260927 15:07:38.875373 11530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
Sep 27 15:07:38 dist-test-slave-hhpq krb5kdc[11291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521658, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:38.885159 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 11530
I20260927 15:07:38.885290 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:07:38.885600 7239 external_mini_cluster.cc:1428] Setting key ef63ff0ae93c9d6a1296ea70acb61cf8
I20260927 15:07:38.889385 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:38.877209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:42247 (local address 127.7.17.254:46873)
0927 15:07:38.877366 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:38.877371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:38.878328 (+ 957us) server_negotiation.cc:407] Connection header received
0927 15:07:38.879572 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:38.879575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:38.879625 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:38.879702 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:38.880997 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:38.881546 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:38.882700 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:38.882918 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:38.882994 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:38.885574 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:38.885578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:38.885581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:38.885610 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:38.887083 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:38.887601 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:38.887605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:38.887607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:38.887706 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:38.888091 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:38.888094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:38.888096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:38.888320 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:38.888417 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:38.889052 (+ 635us) server_negotiation.cc:299] Negotiation successful
0927 15:07:38.889219 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:07:38.890280 11660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:38.877663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46873 (local address 127.7.17.194:42247)
0927 15:07:38.878156 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:38.878202 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:38.879369 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:38.879870 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:38.879877 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:38.880296 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:38.880806 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:38.880821 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:38.881671 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:38.881674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:38.882585 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:38.882595 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:38.883046 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:38.883049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:38.883201 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:38.883832 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:38.883849 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:38.885398 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:38.887203 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:38.887210 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:38.887228 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:38.887494 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:38.887826 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:38.887829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:38.887831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:38.887999 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:38.888442 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:38.888448 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:38.888754 (+ 306us) client_negotiation.cc:769] Sending connection context
0927 15:07:38.888983 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 15:07:38.889347 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":117,"threads_started":1}
I20260927 15:07:38.891480 11658 heartbeater.cc:347] Connected to a master server at 127.7.17.254:46873
I20260927 15:07:38.891731 11658 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:38.892223 11658 heartbeater.cc:511] Master 127.7.17.254:46873 requested a full tablet report, sending...
I20260927 15:07:38.893447 11334 ts_manager.cc:194] Registered new tserver with Master: 273358c3d1594ae49cb107dedeeda600 (127.7.17.194:45751)
I20260927 15:07:38.894040 11334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:42247
WARNING: no policy specified for kudu/127.7.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
I20260927 15:07:38.912820 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46873
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:39.028496 11663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:39.028787 11663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:39.028861 11663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:39.033844 11663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:39.033929 11663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:39.033994 11663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:07:39.039669 11663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:39.040872 11663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:39.041944 11663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:39.052363 11669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:39.052363 11668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:39.052363 11671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:39.052769 11663 server_base.cc:1034] running on GCE node
I20260927 15:07:39.053268 11663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:39.053972 11663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:39.055133 11663 hybrid_clock.cc:648] HybridClock initialized: now 1790521659055090 us; error 56 us; skew 500 ppm
Sep 27 15:07:39 dist-test-slave-hhpq krb5kdc[11291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521659, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:39.057934 11663 init.cc:377] Logged in from keytab as kudu/127.7.17.195@KRBTEST.COM (short username kudu)
I20260927 15:07:39.058758 11663 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:39.058825 11663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:39.059039 11663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:39.061184 11663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "d61334b1882e43eb93a6135b7d8eb82c"
format_stamp: "Formatted at 2026-09-27 15:07:39 on dist-test-slave-hhpq"
server_key: "a5701b50d62132da96ce53718cf84ff1"
server_key_iv: "6250c6c3d370942ee5041e8520146b6b"
server_key_version: "encryptionkey@0"
I20260927 15:07:39.061790 11663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "d61334b1882e43eb93a6135b7d8eb82c"
format_stamp: "Formatted at 2026-09-27 15:07:39 on dist-test-slave-hhpq"
server_key: "a5701b50d62132da96ce53718cf84ff1"
server_key_iv: "6250c6c3d370942ee5041e8520146b6b"
server_key_version: "encryptionkey@0"
I20260927 15:07:39.065739 11663 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:07:39.068274 11677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:39.069737 11663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:39.069864 11663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "d61334b1882e43eb93a6135b7d8eb82c"
format_stamp: "Formatted at 2026-09-27 15:07:39 on dist-test-slave-hhpq"
server_key: "a5701b50d62132da96ce53718cf84ff1"
server_key_iv: "6250c6c3d370942ee5041e8520146b6b"
server_key_version: "encryptionkey@0"
I20260927 15:07:39.069975 11663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:39.083097 11663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:39.086516 11663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:39.086724 11663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:39.087453 11663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:39.088308 11663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:39.088357 11663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:39.088397 11663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:39.088411 11663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:39.097035 11663 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:34515
I20260927 15:07:39.097074 11790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:34515 every 8 connection(s)
I20260927 15:07:39.097642 11663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:39.099241 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 11663
I20260927 15:07:39.099362 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:07:39.099839 7239 external_mini_cluster.cc:1428] Setting key 8f5a317afc0b18f0bce4795ba6d265db
Sep 27 15:07:39 dist-test-slave-hhpq krb5kdc[11291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521659, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:39.113692 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:39.099586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:42573 (local address 127.7.17.254:46873)
0927 15:07:39.099738 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:39.099742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:39.100924 (+ 1182us) server_negotiation.cc:407] Connection header received
0927 15:07:39.102285 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:39.102287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:39.102341 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:39.102422 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:39.104007 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.104588 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.106281 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.106521 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.106605 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:39.109176 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:39.109180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:39.109183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:39.109213 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:39.111228 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:39.111776 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:39.111780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:39.111781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:39.111861 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:39.112315 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:39.112318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:39.112319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:39.112520 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:39.112603 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:39.113396 (+ 793us) server_negotiation.cc:299] Negotiation successful
0927 15:07:39.113527 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:07:39.114521 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:39.100125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46873 (local address 127.7.17.195:42573)
0927 15:07:39.100746 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:39.100807 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:39.102091 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:39.102642 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:39.102652 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:39.103162 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:39.103837 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:39.103857 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.104715 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.104718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:39.106154 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:39.106165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.106659 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.106662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:39.106821 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:39.107408 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:39.107425 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:39.108989 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:39.111364 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:39.111371 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:39.111390 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:39.111663 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:39.111944 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:39.111948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:39.111950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:39.112141 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:39.112619 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:39.112626 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:39.112990 (+ 364us) client_negotiation.cc:769] Sending connection context
0927 15:07:39.113202 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 15:07:39.113554 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":140,"threads_started":1}
I20260927 15:07:39.115566 11791 heartbeater.cc:347] Connected to a master server at 127.7.17.254:46873
I20260927 15:07:39.115816 11791 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:39.116333 11791 heartbeater.cc:511] Master 127.7.17.254:46873 requested a full tablet report, sending...
I20260927 15:07:39.117399 11334 ts_manager.cc:194] Registered new tserver with Master: d61334b1882e43eb93a6135b7d8eb82c (127.7.17.195:34515)
I20260927 15:07:39.118009 11334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:42573
I20260927 15:07:39.125927 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:07:39.135735 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:39.127871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44956 (local address 127.7.17.254:46873)
0927 15:07:39.128001 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:39.128006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:39.128198 (+ 192us) server_negotiation.cc:407] Connection header received
0927 15:07:39.128269 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:39.128271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:39.128312 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:39.128378 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:39.129059 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.129671 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.130587 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.130827 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.130918 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:39.132246 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:39.132249 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:39.132251 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:39.132276 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:39.133814 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:39.134330 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:39.134334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:39.134335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:39.134423 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:39.134741 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:39.134744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:39.134746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:39.134997 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:39.135116 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:39.135468 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 15:07:39.135576 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:07:39.138922 11334 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:07:39.142030 11334 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:39.156950 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:39.150846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:34515 (local address 127.0.0.1:60538)
0927 15:07:39.151678 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:39.151730 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:39.151854 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:39.152917 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:39.152919 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:39.152932 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:39.153096 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:39.153103 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.154572 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.154575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:39.155517 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:39.155526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.156329 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.156333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:39.156448 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:39.156489 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 15:07:39.156601 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 15:07:39.156740 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":68,"thread_start_us":84,"threads_started":1}
I20260927 15:07:39.156953 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:39.152053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60538 (local address 127.7.17.195:34515)
0927 15:07:39.152466 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:39.152475 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:39.152510 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:07:39.152585 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:39.152590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:39.152748 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:39.153047 (+ 299us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:39.153203 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.154418 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.155644 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.156207 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.156303 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:39.156453 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:39.156642 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 15:07:39.156759 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":118,"threads_started":1}
I20260927 15:07:39.157366 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:39.151172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:44699 (local address 127.0.0.1:54452)
0927 15:07:39.151781 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:39.151792 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:39.151854 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:39.152811 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:39.152817 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:39.152865 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:39.153054 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:39.153063 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.154146 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.154150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:39.155380 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:39.155389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.156524 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.156528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:39.156604 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:39.156624 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 15:07:39.156679 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 15:07:39.156739 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":22,"spinlock_wait_cycles":8960,"thread_start_us":97,"threads_started":1}
I20260927 15:07:39.157581 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:39.151857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54452 (local address 127.7.17.193:44699)
0927 15:07:39.152121 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:39.152127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:39.152152 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:07:39.152229 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:39.152234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:39.152457 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:39.152609 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:39.153190 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.154020 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.155490 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.156322 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.156422 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:39.156545 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:39.156849 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 15:07:39.156946 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":98,"threads_started":1}
I20260927 15:07:39.158320 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:39.150810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:45751 (local address 127.0.0.1:44316)
0927 15:07:39.151679 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:39.151734 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:39.151854 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:39.152863 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:39.152866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:39.152888 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:39.153072 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:39.153080 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.154572 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.154575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:39.156568 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:39.156578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.157942 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.157946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:39.158034 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:39.158067 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:07:39.158113 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 15:07:39.158194 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":937,"thread_start_us":98,"threads_started":1}
I20260927 15:07:39.158931 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:39.150917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44316 (local address 127.7.17.194:45751)
0927 15:07:39.152325 (+ 1408us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:39.152342 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:39.152363 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:07:39.152425 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:39.152430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:39.152701 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:39.152889 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:39.153204 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.154426 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.156702 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.157800 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.158488 (+ 688us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:39.158642 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:39.158697 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 15:07:39.158793 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1314,"thread_start_us":108,"threads_started":1}
I20260927 15:07:39.160418 11725 tablet_service.cc:1467] Processing CreateTablet for tablet c62734daa5b543b0805ffa8e08e83434 (DEFAULT_TABLE table=test-table [id=c2edc0ee55f24a069e8ad9b24ff07323]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:39.160418 11459 tablet_service.cc:1467] Processing CreateTablet for tablet c62734daa5b543b0805ffa8e08e83434 (DEFAULT_TABLE table=test-table [id=c2edc0ee55f24a069e8ad9b24ff07323]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:39.161659 11459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c62734daa5b543b0805ffa8e08e83434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:39.161671 11725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c62734daa5b543b0805ffa8e08e83434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:39.161586 11592 tablet_service.cc:1467] Processing CreateTablet for tablet c62734daa5b543b0805ffa8e08e83434 (DEFAULT_TABLE table=test-table [id=c2edc0ee55f24a069e8ad9b24ff07323]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:39.162623 11592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c62734daa5b543b0805ffa8e08e83434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:39.169505 11809 tablet_bootstrap.cc:492] T c62734daa5b543b0805ffa8e08e83434 P 2a96bccb9df940199cbc9870e732e96a: Bootstrap starting.
I20260927 15:07:39.169778 11810 tablet_bootstrap.cc:492] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c: Bootstrap starting.
I20260927 15:07:39.171553 11810 tablet_bootstrap.cc:654] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:39.171564 11809 tablet_bootstrap.cc:654] T c62734daa5b543b0805ffa8e08e83434 P 2a96bccb9df940199cbc9870e732e96a: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:39.172282 11809 log.cc:824] T c62734daa5b543b0805ffa8e08e83434 P 2a96bccb9df940199cbc9870e732e96a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:39.172574 11810 log.cc:824] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:39.173348 11808 tablet_bootstrap.cc:492] T c62734daa5b543b0805ffa8e08e83434 P 273358c3d1594ae49cb107dedeeda600: Bootstrap starting.
I20260927 15:07:39.174866 11810 tablet_bootstrap.cc:492] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c: No bootstrap required, opened a new log
I20260927 15:07:39.175067 11810 ts_tablet_manager.cc:1397] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:07:39.175323 11809 tablet_bootstrap.cc:492] T c62734daa5b543b0805ffa8e08e83434 P 2a96bccb9df940199cbc9870e732e96a: No bootstrap required, opened a new log
I20260927 15:07:39.175504 11809 ts_tablet_manager.cc:1397] T c62734daa5b543b0805ffa8e08e83434 P 2a96bccb9df940199cbc9870e732e96a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 15:07:39.175746 11808 tablet_bootstrap.cc:654] T c62734daa5b543b0805ffa8e08e83434 P 273358c3d1594ae49cb107dedeeda600: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:39.177021 11808 log.cc:824] T c62734daa5b543b0805ffa8e08e83434 P 273358c3d1594ae49cb107dedeeda600: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:39.178435 11810 raft_consensus.cc:348] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "273358c3d1594ae49cb107dedeeda600" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 45751 } } peers { permanent_uuid: "d61334b1882e43eb93a6135b7d8eb82c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 34515 } } peers { permanent_uuid: "2a96bccb9df940199cbc9870e732e96a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 44699 } }
I20260927 15:07:39.178762 11810 raft_consensus.cc:374] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:39.178820 11810 raft_consensus.cc:729] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d61334b1882e43eb93a6135b7d8eb82c, State: Initialized, Role: FOLLOWER
I20260927 15:07:39.179378 11808 tablet_bootstrap.cc:492] T c62734daa5b543b0805ffa8e08e83434 P 273358c3d1594ae49cb107dedeeda600: No bootstrap required, opened a new log
I20260927 15:07:39.179332 11810 consensus_queue.cc:260] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "273358c3d1594ae49cb107dedeeda600" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 45751 } } peers { permanent_uuid: "d61334b1882e43eb93a6135b7d8eb82c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 34515 } } peers { permanent_uuid: "2a96bccb9df940199cbc9870e732e96a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 44699 } }
I20260927 15:07:39.179601 11808 ts_tablet_manager.cc:1397] T c62734daa5b543b0805ffa8e08e83434 P 273358c3d1594ae49cb107dedeeda600: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260927 15:07:39.180083 11791 heartbeater.cc:503] Master 127.7.17.254:46873 was elected leader, sending a full tablet report...
I20260927 15:07:39.180431 11810 ts_tablet_manager.cc:1428] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 15:07:39.179569 11809 raft_consensus.cc:348] T c62734daa5b543b0805ffa8e08e83434 P 2a96bccb9df940199cbc9870e732e96a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "273358c3d1594ae49cb107dedeeda600" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 45751 } } peers { permanent_uuid: "d61334b1882e43eb93a6135b7d8eb82c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 34515 } } peers { permanent_uuid: "2a96bccb9df940199cbc9870e732e96a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 44699 } }
I20260927 15:07:39.181015 11809 raft_consensus.cc:374] T c62734daa5b543b0805ffa8e08e83434 P 2a96bccb9df940199cbc9870e732e96a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:39.181077 11809 raft_consensus.cc:729] T c62734daa5b543b0805ffa8e08e83434 P 2a96bccb9df940199cbc9870e732e96a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a96bccb9df940199cbc9870e732e96a, State: Initialized, Role: FOLLOWER
I20260927 15:07:39.181592 11809 consensus_queue.cc:260] T c62734daa5b543b0805ffa8e08e83434 P 2a96bccb9df940199cbc9870e732e96a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "273358c3d1594ae49cb107dedeeda600" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 45751 } } peers { permanent_uuid: "d61334b1882e43eb93a6135b7d8eb82c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 34515 } } peers { permanent_uuid: "2a96bccb9df940199cbc9870e732e96a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 44699 } }
I20260927 15:07:39.182268 11525 heartbeater.cc:503] Master 127.7.17.254:46873 was elected leader, sending a full tablet report...
I20260927 15:07:39.183374 11808 raft_consensus.cc:348] T c62734daa5b543b0805ffa8e08e83434 P 273358c3d1594ae49cb107dedeeda600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "273358c3d1594ae49cb107dedeeda600" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 45751 } } peers { permanent_uuid: "d61334b1882e43eb93a6135b7d8eb82c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 34515 } } peers { permanent_uuid: "2a96bccb9df940199cbc9870e732e96a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 44699 } }
I20260927 15:07:39.183668 11808 raft_consensus.cc:374] T c62734daa5b543b0805ffa8e08e83434 P 273358c3d1594ae49cb107dedeeda600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:39.183729 11808 raft_consensus.cc:729] T c62734daa5b543b0805ffa8e08e83434 P 273358c3d1594ae49cb107dedeeda600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273358c3d1594ae49cb107dedeeda600, State: Initialized, Role: FOLLOWER
I20260927 15:07:39.183807 11809 ts_tablet_manager.cc:1428] T c62734daa5b543b0805ffa8e08e83434 P 2a96bccb9df940199cbc9870e732e96a: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 15:07:39.184149 11808 consensus_queue.cc:260] T c62734daa5b543b0805ffa8e08e83434 P 273358c3d1594ae49cb107dedeeda600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "273358c3d1594ae49cb107dedeeda600" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 45751 } } peers { permanent_uuid: "d61334b1882e43eb93a6135b7d8eb82c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 34515 } } peers { permanent_uuid: "2a96bccb9df940199cbc9870e732e96a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 44699 } }
I20260927 15:07:39.185341 11808 ts_tablet_manager.cc:1428] T c62734daa5b543b0805ffa8e08e83434 P 273358c3d1594ae49cb107dedeeda600: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260927 15:07:39.185693 11658 heartbeater.cc:503] Master 127.7.17.254:46873 was elected leader, sending a full tablet report...
I20260927 15:07:39.290926 11814 raft_consensus.cc:482] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:39.291225 11814 raft_consensus.cc:504] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "273358c3d1594ae49cb107dedeeda600" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 45751 } } peers { permanent_uuid: "d61334b1882e43eb93a6135b7d8eb82c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 34515 } } peers { permanent_uuid: "2a96bccb9df940199cbc9870e732e96a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 44699 } }
I20260927 15:07:39.292424 11814 leader_election.cc:288] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 273358c3d1594ae49cb107dedeeda600 (127.7.17.194:45751), 2a96bccb9df940199cbc9870e732e96a (127.7.17.193:44699)
I20260927 15:07:39.296551 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:39.292874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:47651 (local address 127.7.17.194:45751)
0927 15:07:39.292993 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:39.292997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:39.293015 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:07:39.293182 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:39.293186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:39.293240 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:39.293320 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:39.293666 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.294346 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.295524 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.295955 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.296053 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:39.296095 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:39.296342 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 15:07:39.296421 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:07:39.296551 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:39.292975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:58957 (local address 127.7.17.193:44699)
0927 15:07:39.293097 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:39.293101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:39.293369 (+ 268us) server_negotiation.cc:407] Connection header received
0927 15:07:39.293499 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:39.293502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:39.293550 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:39.293623 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:39.293983 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.294537 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.295610 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.296058 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.296172 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:39.296220 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:39.296376 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 15:07:39.296441 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:07:39.296552 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:39.292727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:45751 (local address 127.7.17.195:47651)
0927 15:07:39.292907 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:39.292923 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:39.293044 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:39.293345 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:39.293349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:39.293397 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:39.293509 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:39.293515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.294477 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.294481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:39.295389 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:39.295399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.296084 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.296088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:39.296202 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:39.296224 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:07:39.296297 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 15:07:39.296370 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":21,"spinlock_wait_cycles":2304}
I20260927 15:07:39.296571 11817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:39.292727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:44699 (local address 127.7.17.195:58957)
0927 15:07:39.293279 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:39.293292 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:39.293407 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:39.293666 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:39.293669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:39.293688 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:39.293837 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:39.293844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.294654 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.294657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:39.295477 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:39.295488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.296177 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.296180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:39.296254 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:39.296270 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 15:07:39.296355 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 15:07:39.296420 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":160,"spinlock_wait_cycles":37376,"thread_start_us":139,"threads_started":1}
I20260927 15:07:39.297241 11479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62734daa5b543b0805ffa8e08e83434" candidate_uuid: "d61334b1882e43eb93a6135b7d8eb82c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a96bccb9df940199cbc9870e732e96a" is_pre_election: true
I20260927 15:07:39.297286 11612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62734daa5b543b0805ffa8e08e83434" candidate_uuid: "d61334b1882e43eb93a6135b7d8eb82c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273358c3d1594ae49cb107dedeeda600" is_pre_election: true
I20260927 15:07:39.297602 11479 raft_consensus.cc:2454] T c62734daa5b543b0805ffa8e08e83434 P 2a96bccb9df940199cbc9870e732e96a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d61334b1882e43eb93a6135b7d8eb82c in term 0.
I20260927 15:07:39.297602 11612 raft_consensus.cc:2454] T c62734daa5b543b0805ffa8e08e83434 P 273358c3d1594ae49cb107dedeeda600 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d61334b1882e43eb93a6135b7d8eb82c in term 0.
I20260927 15:07:39.298151 11678 leader_election.cc:302] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a96bccb9df940199cbc9870e732e96a, d61334b1882e43eb93a6135b7d8eb82c; no voters:
I20260927 15:07:39.298426 11814 raft_consensus.cc:2781] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:39.298537 11814 raft_consensus.cc:482] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:39.298590 11814 raft_consensus.cc:3037] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:39.299840 11814 raft_consensus.cc:504] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "273358c3d1594ae49cb107dedeeda600" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 45751 } } peers { permanent_uuid: "d61334b1882e43eb93a6135b7d8eb82c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 34515 } } peers { permanent_uuid: "2a96bccb9df940199cbc9870e732e96a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 44699 } }
I20260927 15:07:39.300207 11814 leader_election.cc:288] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c [CANDIDATE]: Term 1 election: Requested vote from peers 273358c3d1594ae49cb107dedeeda600 (127.7.17.194:45751), 2a96bccb9df940199cbc9870e732e96a (127.7.17.193:44699)
I20260927 15:07:39.300647 11479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62734daa5b543b0805ffa8e08e83434" candidate_uuid: "d61334b1882e43eb93a6135b7d8eb82c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a96bccb9df940199cbc9870e732e96a"
I20260927 15:07:39.300652 11612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c62734daa5b543b0805ffa8e08e83434" candidate_uuid: "d61334b1882e43eb93a6135b7d8eb82c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273358c3d1594ae49cb107dedeeda600"
I20260927 15:07:39.300776 11479 raft_consensus.cc:3037] T c62734daa5b543b0805ffa8e08e83434 P 2a96bccb9df940199cbc9870e732e96a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:39.300782 11612 raft_consensus.cc:3037] T c62734daa5b543b0805ffa8e08e83434 P 273358c3d1594ae49cb107dedeeda600 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:39.302006 11479 raft_consensus.cc:2454] T c62734daa5b543b0805ffa8e08e83434 P 2a96bccb9df940199cbc9870e732e96a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d61334b1882e43eb93a6135b7d8eb82c in term 1.
I20260927 15:07:39.302017 11612 raft_consensus.cc:2454] T c62734daa5b543b0805ffa8e08e83434 P 273358c3d1594ae49cb107dedeeda600 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d61334b1882e43eb93a6135b7d8eb82c in term 1.
I20260927 15:07:39.302383 11681 leader_election.cc:302] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 273358c3d1594ae49cb107dedeeda600, 2a96bccb9df940199cbc9870e732e96a, d61334b1882e43eb93a6135b7d8eb82c; no voters:
I20260927 15:07:39.302640 11814 raft_consensus.cc:2781] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:39.302878 11814 raft_consensus.cc:686] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c [term 1 LEADER]: Becoming Leader. State: Replica: d61334b1882e43eb93a6135b7d8eb82c, State: Running, Role: LEADER
I20260927 15:07:39.303241 11814 consensus_queue.cc:237] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "273358c3d1594ae49cb107dedeeda600" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 45751 } } peers { permanent_uuid: "d61334b1882e43eb93a6135b7d8eb82c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 34515 } } peers { permanent_uuid: "2a96bccb9df940199cbc9870e732e96a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 44699 } }
I20260927 15:07:39.306473 11335 catalog_manager.cc:5495] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c reported cstate change: term changed from 0 to 1, leader changed from <none> to d61334b1882e43eb93a6135b7d8eb82c (127.7.17.195). New cstate: current_term: 1 leader_uuid: "d61334b1882e43eb93a6135b7d8eb82c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "273358c3d1594ae49cb107dedeeda600" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 45751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d61334b1882e43eb93a6135b7d8eb82c" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 34515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a96bccb9df940199cbc9870e732e96a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 44699 } health_report { overall_health: UNKNOWN } } }
I20260927 15:07:39.347290 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:39.343864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60554 (local address 127.7.17.195:34515)
0927 15:07:39.343990 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:39.343995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:39.344009 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:07:39.344119 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:39.344121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:39.344185 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:39.344289 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:39.344521 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.345082 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.345923 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:39.346128 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:39.346222 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:39.346492 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:39.346914 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:39.347025 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:39.347081 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 15:07:39.347139 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 15:07:39.348559 11792 tablet.cc:2369] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:07:39.353813 11612 raft_consensus.cc:1261] T c62734daa5b543b0805ffa8e08e83434 P 273358c3d1594ae49cb107dedeeda600 [term 1 FOLLOWER]: Refusing update from remote peer d61334b1882e43eb93a6135b7d8eb82c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:39.353814 11479 raft_consensus.cc:1261] T c62734daa5b543b0805ffa8e08e83434 P 2a96bccb9df940199cbc9870e732e96a [term 1 FOLLOWER]: Refusing update from remote peer d61334b1882e43eb93a6135b7d8eb82c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:39.354754 11818 consensus_queue.cc:1035] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c [LEADER]: Connected to new peer: Peer: permanent_uuid: "273358c3d1594ae49cb107dedeeda600" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 45751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:39.355049 11814 consensus_queue.cc:1035] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a96bccb9df940199cbc9870e732e96a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 44699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:39.363099 11823 mvcc.cc:205] Tried to move back new op lower bound from 7333976716703920128 to 7333976716508807168. Current Snapshot: MvccSnapshot[applied={T|T < 7333976716703920128}]
I20260927 15:07:39.371816 11335 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:39.371991 11335 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:39.374593 11335 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5999c65ba7ab498a9cb7c8f8cbd7266c: Sending DeleteTablet for 3 replicas of tablet c62734daa5b543b0805ffa8e08e83434
I20260927 15:07:39.375418 11592 tablet_service.cc:1514] Processing DeleteTablet for tablet c62734daa5b543b0805ffa8e08e83434 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:39 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:44316
I20260927 15:07:39.376042 11830 tablet_replica.cc:330] T c62734daa5b543b0805ffa8e08e83434 P 273358c3d1594ae49cb107dedeeda600: stopping tablet replica
I20260927 15:07:39.376894 11830 raft_consensus.cc:2229] T c62734daa5b543b0805ffa8e08e83434 P 273358c3d1594ae49cb107dedeeda600 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:39.377229 11830 raft_consensus.cc:2258] T c62734daa5b543b0805ffa8e08e83434 P 273358c3d1594ae49cb107dedeeda600 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:39.378811 11830 ts_tablet_manager.cc:1905] T c62734daa5b543b0805ffa8e08e83434 P 273358c3d1594ae49cb107dedeeda600: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:39.381026 11725 tablet_service.cc:1514] Processing DeleteTablet for tablet c62734daa5b543b0805ffa8e08e83434 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:39 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:60538
I20260927 15:07:39.381417 11459 tablet_service.cc:1514] Processing DeleteTablet for tablet c62734daa5b543b0805ffa8e08e83434 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:39 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:54452
I20260927 15:07:39.381485 11830 ts_tablet_manager.cc:1918] T c62734daa5b543b0805ffa8e08e83434 P 273358c3d1594ae49cb107dedeeda600: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:39.381565 11830 log.cc:1196] T c62734daa5b543b0805ffa8e08e83434 P 273358c3d1594ae49cb107dedeeda600: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-1/wal/wals/c62734daa5b543b0805ffa8e08e83434
I20260927 15:07:39.381819 11830 ts_tablet_manager.cc:1939] T c62734daa5b543b0805ffa8e08e83434 P 273358c3d1594ae49cb107dedeeda600: Deleting consensus metadata
I20260927 15:07:39.382421 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 11397
I20260927 15:07:39.384016 11322 catalog_manager.cc:4841] TS 273358c3d1594ae49cb107dedeeda600 (127.7.17.194:45751): tablet c62734daa5b543b0805ffa8e08e83434 (table test-table [id=c2edc0ee55f24a069e8ad9b24ff07323]) successfully deleted
I20260927 15:07:39.389084 11831 tablet_replica.cc:330] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c: stopping tablet replica
I20260927 15:07:39.389474 11831 raft_consensus.cc:2229] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:07:39.389855 11831 raft_consensus.cc:2258] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 15:07:39.390187 11319 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.17.193:44699 (error 108)
I20260927 15:07:39.391029 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 11530
I20260927 15:07:39.391664 11831 ts_tablet_manager.cc:1905] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:39.393846 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:39.393494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:44699 (local address 127.0.0.1:54466)
0927 15:07:39.393688 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:39.393759 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.193:44699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260927 15:07:39.394119 11319 catalog_manager.cc:4584] TS 2a96bccb9df940199cbc9870e732e96a (127.7.17.193:44699): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c62734daa5b543b0805ffa8e08e83434: Network error: Client connection negotiation failed: client connection to 127.7.17.193:44699: connect: Connection refused (error 111)
I20260927 15:07:39.394217 11831 ts_tablet_manager.cc:1918] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:39.394302 11831 log.cc:1196] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/c62734daa5b543b0805ffa8e08e83434
I20260927 15:07:39.394565 11831 ts_tablet_manager.cc:1939] T c62734daa5b543b0805ffa8e08e83434 P d61334b1882e43eb93a6135b7d8eb82c: Deleting consensus metadata
I20260927 15:07:39.395658 11320 catalog_manager.cc:4841] TS d61334b1882e43eb93a6135b7d8eb82c (127.7.17.195:34515): tablet c62734daa5b543b0805ffa8e08e83434 (table test-table [id=c2edc0ee55f24a069e8ad9b24ff07323]) successfully deleted
I20260927 15:07:39.397677 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 11663
I20260927 15:07:39.402869 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 11304
2026-09-27T15:07:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2291 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T15:07:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:39Z Disabled control of system clock
I20260927 15:07:39.431564 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:42887
--webserver_interface=127.7.17.254
--webserver_port=0
--builtin_ntp_servers=127.7.17.212:44357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:42887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 15:07:39.542161 11837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:39.542459 11837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:39.542496 11837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:39.546715 11837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:39.546785 11837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:39.546799 11837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:39.546815 11837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:39.546829 11837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:39.551880 11837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:44357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:42887
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:42887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:39.553071 11837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:39.554548 11837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:39.563339 11843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:39.563402 11837 server_base.cc:1034] running on GCE node
W20260927 15:07:39.563350 11845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:39.563350 11842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:39.564118 11837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:39.565368 11837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:39.566581 11837 hybrid_clock.cc:648] HybridClock initialized: now 1790521659566563 us; error 30 us; skew 500 ppm
I20260927 15:07:39.568559 11837 webserver.cc:460] Webserver started at http://127.7.17.254:37717/ using document root <none> and password file <none>
I20260927 15:07:39.569181 11837 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:39.569253 11837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:39.569434 11837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:39.571511 11837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "1cd1bbe6742049ec8e7a83b8e7948e7c"
format_stamp: "Formatted at 2026-09-27 15:07:39 on dist-test-slave-hhpq"
server_key: "bc9c4762ad11f97de616e0fa40d11058"
server_key_iv: "5058c6cc3feb30dde8646182f3d842ab"
server_key_version: "encryptionkey@0"
I20260927 15:07:39.572039 11837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "1cd1bbe6742049ec8e7a83b8e7948e7c"
format_stamp: "Formatted at 2026-09-27 15:07:39 on dist-test-slave-hhpq"
server_key: "bc9c4762ad11f97de616e0fa40d11058"
server_key_iv: "5058c6cc3feb30dde8646182f3d842ab"
server_key_version: "encryptionkey@0"
I20260927 15:07:39.575954 11837 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260927 15:07:39.578668 11851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:39.580092 11837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:07:39.580282 11837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "1cd1bbe6742049ec8e7a83b8e7948e7c"
format_stamp: "Formatted at 2026-09-27 15:07:39 on dist-test-slave-hhpq"
server_key: "bc9c4762ad11f97de616e0fa40d11058"
server_key_iv: "5058c6cc3feb30dde8646182f3d842ab"
server_key_version: "encryptionkey@0"
I20260927 15:07:39.580397 11837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:39.585253 11837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:39.585897 11837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:39.586058 11837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:39.592869 11837 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:42887
I20260927 15:07:39.592875 11903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:42887 every 8 connection(s)
I20260927 15:07:39.593832 11837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:39.597044 11904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:39.597430 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 11837
I20260927 15:07:39.597533 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:07:39.597792 7239 external_mini_cluster.cc:1428] Setting key 96b66d48873bd357cc3ccad06afb3a72
I20260927 15:07:39.603772 11904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c: Bootstrap starting.
I20260927 15:07:39.603967 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:39.599197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42998 (local address 127.7.17.254:42887)
0927 15:07:39.599919 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:39.599938 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:39.599985 (+ 47us) server_negotiation.cc:407] Connection header received
0927 15:07:39.600813 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:39.600860 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:39.601334 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:39.601718 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:39.602434 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:39.602440 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:39.602456 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:39.602470 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:39.602544 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:39.602566 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:39.602712 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:39.602787 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 15:07:39.603036 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":112,"threads_started":1}
I20260927 15:07:39.606101 11904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:39.606819 11904 log.cc:824] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:39.609045 11904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c: No bootstrap required, opened a new log
I20260927 15:07:39.612762 11904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd1bbe6742049ec8e7a83b8e7948e7c" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42887 } }
I20260927 15:07:39.613070 11904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:39.613138 11904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd1bbe6742049ec8e7a83b8e7948e7c, State: Initialized, Role: FOLLOWER
I20260927 15:07:39.613641 11904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd1bbe6742049ec8e7a83b8e7948e7c" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42887 } }
I20260927 15:07:39.613786 11904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:39.613852 11904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:39.613950 11904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:39.614905 11904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd1bbe6742049ec8e7a83b8e7948e7c" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42887 } }
I20260927 15:07:39.615262 11904 leader_election.cc:302] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cd1bbe6742049ec8e7a83b8e7948e7c; no voters:
I20260927 15:07:39.615595 11904 leader_election.cc:288] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:39.615720 11909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:39.615932 11909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c [term 1 LEADER]: Becoming Leader. State: Replica: 1cd1bbe6742049ec8e7a83b8e7948e7c, State: Running, Role: LEADER
I20260927 15:07:39.616273 11909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd1bbe6742049ec8e7a83b8e7948e7c" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42887 } }
I20260927 15:07:39.616900 11904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:39.617975 11910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cd1bbe6742049ec8e7a83b8e7948e7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd1bbe6742049ec8e7a83b8e7948e7c" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42887 } } }
I20260927 15:07:39.617985 11911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cd1bbe6742049ec8e7a83b8e7948e7c. Latest consensus state: current_term: 1 leader_uuid: "1cd1bbe6742049ec8e7a83b8e7948e7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd1bbe6742049ec8e7a83b8e7948e7c" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42887 } } }
I20260927 15:07:39.618111 11910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:39.618111 11911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:39.618618 11918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:39.621484 11918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:39.627203 11918 catalog_manager.cc:1347] Generated new cluster ID: 2bfb1ec8b12c410687495c52f923c129
I20260927 15:07:39.627280 11918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:39.634927 11918 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:39.635291 11918 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:39.639204 11918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c: Generated new TSK 0
I20260927 15:07:39.639904 11918 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:07:39.644081 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:42887
--builtin_ntp_servers=127.7.17.212:44357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 15:07:39.758029 11928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:39.758327 11928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:39.758366 11928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:39.762599 11928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:39.762668 11928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:39.762738 11928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:07:39.767729 11928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:44357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:42887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:39.769114 11928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:39.770660 11928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:39.780762 11933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:39.780912 11928 server_base.cc:1034] running on GCE node
W20260927 15:07:39.780776 11934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:39.780736 11936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:39.781492 11928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:39.782188 11928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:39.783356 11928 hybrid_clock.cc:648] HybridClock initialized: now 1790521659783321 us; error 48 us; skew 500 ppm
I20260927 15:07:39.785333 11928 webserver.cc:460] Webserver started at http://127.7.17.193:36939/ using document root <none> and password file <none>
I20260927 15:07:39.785917 11928 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:39.785980 11928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:39.786175 11928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:39.788278 11928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "12b87f0dd1b2497e8dfd78d7c9b2e328"
format_stamp: "Formatted at 2026-09-27 15:07:39 on dist-test-slave-hhpq"
server_key: "b2c17e136d54f77b706663a9fe088a04"
server_key_iv: "47ca43ee01c4b59f88d842ede4aab3f8"
server_key_version: "encryptionkey@0"
I20260927 15:07:39.788853 11928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "12b87f0dd1b2497e8dfd78d7c9b2e328"
format_stamp: "Formatted at 2026-09-27 15:07:39 on dist-test-slave-hhpq"
server_key: "b2c17e136d54f77b706663a9fe088a04"
server_key_iv: "47ca43ee01c4b59f88d842ede4aab3f8"
server_key_version: "encryptionkey@0"
I20260927 15:07:39.792622 11928 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:39.795265 11942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:39.796538 11928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:07:39.796659 11928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "12b87f0dd1b2497e8dfd78d7c9b2e328"
format_stamp: "Formatted at 2026-09-27 15:07:39 on dist-test-slave-hhpq"
server_key: "b2c17e136d54f77b706663a9fe088a04"
server_key_iv: "47ca43ee01c4b59f88d842ede4aab3f8"
server_key_version: "encryptionkey@0"
I20260927 15:07:39.796788 11928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:39.801461 11928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:39.802083 11928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:39.802240 11928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:39.802791 11928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:39.803699 11928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:39.803751 11928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:39.803805 11928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:39.803833 11928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:39.811892 11928 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:33147
I20260927 15:07:39.811913 12055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:33147 every 8 connection(s)
I20260927 15:07:39.812803 11928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:39.818913 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:39.814658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:33701 (local address 127.7.17.254:42887)
0927 15:07:39.814824 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:39.814828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:39.815897 (+ 1069us) server_negotiation.cc:407] Connection header received
0927 15:07:39.817284 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:39.817288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:39.817330 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:39.817404 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:39.818544 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:39.818547 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:39.818549 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:39.818561 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:39.818579 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:39.818587 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:39.818661 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:39.818768 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 15:07:39.818805 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:07:39.819789 12058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:39.815060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:42887 (local address 127.7.17.193:33701)
0927 15:07:39.815754 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:39.815797 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:39.817029 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:39.817607 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:39.817615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:39.817727 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:39.818300 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:07:39.818301 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:39.818344 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:07:39.818414 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:39.818656 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:39.818662 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:39.818666 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 15:07:39.818800 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 15:07:39.819009 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":104,"threads_started":1}
I20260927 15:07:39.820503 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 11928
I20260927 15:07:39.820602 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:07:39.820880 7239 external_mini_cluster.cc:1428] Setting key 98eb5439477edd515a4c4983d422a02e
I20260927 15:07:39.821022 12056 heartbeater.cc:347] Connected to a master server at 127.7.17.254:42887
I20260927 15:07:39.821264 12056 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:39.821660 12056 heartbeater.cc:511] Master 127.7.17.254:42887 requested a full tablet report, sending...
I20260927 15:07:39.822794 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:42887
--builtin_ntp_servers=127.7.17.212:44357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 15:07:39.823325 11868 ts_manager.cc:194] Registered new tserver with Master: 12b87f0dd1b2497e8dfd78d7c9b2e328 (127.7.17.193:33147)
W20260927 15:07:39.934470 12059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:39.934756 12059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:39.934796 12059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:39.939024 12059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:39.939087 12059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:39.939146 12059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:07:39.944078 12059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:44357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:42887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:39.945343 12059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:39.946748 12059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:39.956631 12064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:39.956686 12067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:39.956699 12059 server_base.cc:1034] running on GCE node
W20260927 15:07:39.956919 12065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:39.957336 12059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:39.957952 12059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:39.959133 12059 hybrid_clock.cc:648] HybridClock initialized: now 1790521659959103 us; error 45 us; skew 500 ppm
I20260927 15:07:39.961066 12059 webserver.cc:460] Webserver started at http://127.7.17.194:33305/ using document root <none> and password file <none>
I20260927 15:07:39.961642 12059 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:39.961709 12059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:39.961892 12059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:39.963986 12059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "ae2f44963134453fa55077d73fe9ae9d"
format_stamp: "Formatted at 2026-09-27 15:07:39 on dist-test-slave-hhpq"
server_key: "b1eb6ffb5b782ec916f5c321fbef125a"
server_key_iv: "731ed8579bb864c4556bbb4acfa7d93c"
server_key_version: "encryptionkey@0"
I20260927 15:07:39.964532 12059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "ae2f44963134453fa55077d73fe9ae9d"
format_stamp: "Formatted at 2026-09-27 15:07:39 on dist-test-slave-hhpq"
server_key: "b1eb6ffb5b782ec916f5c321fbef125a"
server_key_iv: "731ed8579bb864c4556bbb4acfa7d93c"
server_key_version: "encryptionkey@0"
I20260927 15:07:39.968194 12059 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:07:39.970692 12073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:39.972003 12059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:07:39.972100 12059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "ae2f44963134453fa55077d73fe9ae9d"
format_stamp: "Formatted at 2026-09-27 15:07:39 on dist-test-slave-hhpq"
server_key: "b1eb6ffb5b782ec916f5c321fbef125a"
server_key_iv: "731ed8579bb864c4556bbb4acfa7d93c"
server_key_version: "encryptionkey@0"
I20260927 15:07:39.972220 12059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:39.976989 12059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:39.977610 12059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:39.977777 12059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:39.978312 12059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:39.979254 12059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:39.979303 12059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:39.979355 12059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:39.979385 12059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:39.987744 12059 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:38125
I20260927 15:07:39.987763 12186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:38125 every 8 connection(s)
I20260927 15:07:39.988719 12059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:07:39.989622 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 12059
I20260927 15:07:39.989729 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:07:39.989984 7239 external_mini_cluster.cc:1428] Setting key 9bc145d1715204e33cdfe90bd1c53870
I20260927 15:07:39.992125 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:42887
--builtin_ntp_servers=127.7.17.212:44357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 15:07:39.996528 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:39.990686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:35501 (local address 127.7.17.254:42887)
0927 15:07:39.990813 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:39.990816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:39.993479 (+ 2663us) server_negotiation.cc:407] Connection header received
0927 15:07:39.994832 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:39.994836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:39.994884 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:39.994960 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:39.996092 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:39.996096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:39.996099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:39.996112 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:39.996134 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:39.996142 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:39.996221 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:39.996342 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 15:07:39.996387 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:07:39.997447 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:39.991242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:42887 (local address 127.7.17.194:35501)
0927 15:07:39.993335 (+ 2093us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:39.993379 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:39.994596 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:39.995153 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:39.995172 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:39.995288 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:39.995837 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:07:39.995839 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:39.995881 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:07:39.995951 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:39.996211 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:39.996219 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:39.996225 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:07:39.996383 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 15:07:39.996645 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1656,"thread_start_us":107,"threads_started":1}
I20260927 15:07:39.998795 12187 heartbeater.cc:347] Connected to a master server at 127.7.17.254:42887
I20260927 15:07:39.999018 12187 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:39.999450 12187 heartbeater.cc:511] Master 127.7.17.254:42887 requested a full tablet report, sending...
I20260927 15:07:40.000542 11868 ts_manager.cc:194] Registered new tserver with Master: ae2f44963134453fa55077d73fe9ae9d (127.7.17.194:38125)
W20260927 15:07:40.103981 12190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:40.104260 12190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:40.104300 12190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:40.108412 12190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:40.108470 12190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:40.108527 12190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:07:40.113441 12190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:44357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:42887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:40.114472 12190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:40.115818 12190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:40.125504 12195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:40.125547 12196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:40.125528 12198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:40.125775 12190 server_base.cc:1034] running on GCE node
I20260927 15:07:40.126288 12190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:40.126929 12190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:40.128110 12190 hybrid_clock.cc:648] HybridClock initialized: now 1790521660128060 us; error 65 us; skew 500 ppm
I20260927 15:07:40.130153 12190 webserver.cc:460] Webserver started at http://127.7.17.195:41615/ using document root <none> and password file <none>
I20260927 15:07:40.130746 12190 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:40.130813 12190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:40.130997 12190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:40.133170 12190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "d66badaf0f0446bcb8aba0ad8bad2270"
format_stamp: "Formatted at 2026-09-27 15:07:40 on dist-test-slave-hhpq"
server_key: "5d25075827f6a18b0642864cb1738bee"
server_key_iv: "dd9300ea91271004bd88058f70528e3d"
server_key_version: "encryptionkey@0"
I20260927 15:07:40.133736 12190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "d66badaf0f0446bcb8aba0ad8bad2270"
format_stamp: "Formatted at 2026-09-27 15:07:40 on dist-test-slave-hhpq"
server_key: "5d25075827f6a18b0642864cb1738bee"
server_key_iv: "dd9300ea91271004bd88058f70528e3d"
server_key_version: "encryptionkey@0"
I20260927 15:07:40.137861 12190 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:07:40.140667 12204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:40.142058 12190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:40.142194 12190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "d66badaf0f0446bcb8aba0ad8bad2270"
format_stamp: "Formatted at 2026-09-27 15:07:40 on dist-test-slave-hhpq"
server_key: "5d25075827f6a18b0642864cb1738bee"
server_key_iv: "dd9300ea91271004bd88058f70528e3d"
server_key_version: "encryptionkey@0"
I20260927 15:07:40.142333 12190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:40.147059 12190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:40.147673 12190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:40.147830 12190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:40.148394 12190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:40.149420 12190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:40.149469 12190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:40.149523 12190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:40.149549 12190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:40.158123 12190 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:42221
I20260927 15:07:40.158161 12317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:42221 every 8 connection(s)
I20260927 15:07:40.159075 12190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:40.165263 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:40.161001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:56363 (local address 127.7.17.254:42887)
0927 15:07:40.161147 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:40.161152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:40.162324 (+ 1172us) server_negotiation.cc:407] Connection header received
0927 15:07:40.163589 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:40.163593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:40.163638 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:40.163716 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:40.164888 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:40.164892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:40.164894 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:40.164909 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:40.164932 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:40.164943 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:40.165020 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:40.165116 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 15:07:40.165152 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:07:40.166218 12320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:40.161472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:42887 (local address 127.7.17.195:56363)
0927 15:07:40.162172 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:40.162213 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:40.163367 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:40.163891 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:40.163899 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:40.163995 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:40.164558 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:07:40.164559 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:40.164604 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:07:40.164688 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:40.165009 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:40.165014 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:40.165018 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 15:07:40.165154 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 15:07:40.165365 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":118,"threads_started":1}
I20260927 15:07:40.167567 12318 heartbeater.cc:347] Connected to a master server at 127.7.17.254:42887
I20260927 15:07:40.167801 12318 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:40.168188 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 12190
I20260927 15:07:40.168279 12318 heartbeater.cc:511] Master 127.7.17.254:42887 requested a full tablet report, sending...
I20260927 15:07:40.168299 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:07:40.168592 7239 external_mini_cluster.cc:1428] Setting key 770f2d720ddc8ba12c68ac669b59a1c4
I20260927 15:07:40.169543 11867 ts_manager.cc:194] Registered new tserver with Master: d66badaf0f0446bcb8aba0ad8bad2270 (127.7.17.195:42221)
I20260927 15:07:40.170583 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:07:40.173897 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:40.172414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.7.17.254:42887)
0927 15:07:40.172524 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:40.172528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:40.172721 (+ 193us) server_negotiation.cc:407] Connection header received
0927 15:07:40.172866 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:40.172870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:40.172907 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:40.172969 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:40.173578 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:40.173580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:40.173582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:40.173588 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:40.173600 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:40.173606 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:40.173661 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:40.173722 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 15:07:40.173753 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:07:40.176631 11868 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:07:40.179765 11868 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:40.191434 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:40.188145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:42221 (local address 127.0.0.1:57502)
0927 15:07:40.188960 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:40.189012 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:40.189137 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:40.189953 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:40.189959 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:40.190036 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:40.190641 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:07:40.190643 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:40.190665 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:07:40.190737 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:40.191109 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:40.191118 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:40.191124 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:07:40.191248 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 15:07:40.191320 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":78,"threads_started":1}
I20260927 15:07:40.191845 12333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:40.188928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53650 (local address 127.7.17.194:38125)
0927 15:07:40.189628 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:40.189638 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:40.189667 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:07:40.189710 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:40.189714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:40.189839 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:40.189943 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:40.191303 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:40.191309 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:40.191321 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:40.191332 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:40.191379 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:40.191414 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:40.191514 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:40.191614 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 15:07:40.191687 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":113,"threads_started":1}
I20260927 15:07:40.191936 12330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:40.188334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57502 (local address 127.7.17.195:42221)
0927 15:07:40.189472 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:40.189487 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:40.189512 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:07:40.189566 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:40.189572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:40.189779 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:40.189928 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:40.190875 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:40.190881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:40.190894 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:40.190908 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:40.190957 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:40.190999 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:40.191107 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:40.191288 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 15:07:40.191365 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1029,"thread_start_us":97,"threads_started":1}
I20260927 15:07:40.192442 12331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:40.188416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:33147 (local address 127.0.0.1:37746)
0927 15:07:40.189316 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:40.189328 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:40.189406 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:40.189990 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:40.189992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:40.190036 (+ 44us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:40.191776 (+ 1740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:07:40.191777 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:40.191786 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:07:40.191811 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:40.192233 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:40.192236 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:40.192239 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 15:07:40.192293 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 15:07:40.192326 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"spinlock_wait_cycles":4224,"thread_start_us":99,"threads_started":1}
I20260927 15:07:40.191731 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:40.188457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:38125 (local address 127.0.0.1:53650)
0927 15:07:40.189323 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:40.189335 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:40.189406 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:40.190603 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:40.190606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:40.190618 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:40.191111 (+ 493us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:07:40.191112 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:40.191123 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:07:40.191147 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:40.191506 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:40.191509 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:40.191512 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 15:07:40.191590 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 15:07:40.191621 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"spinlock_wait_cycles":4480,"thread_start_us":95,"threads_started":1}
I20260927 15:07:40.192930 12334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:40.188928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37746 (local address 127.7.17.193:33147)
0927 15:07:40.189629 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:40.189637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:40.189675 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:07:40.189717 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:40.189721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:40.189857 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:40.190194 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:40.192017 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:40.192023 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:40.192034 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:40.192047 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:40.192091 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:40.192129 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:40.192620 (+ 491us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:40.192667 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 15:07:40.192752 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":120,"threads_started":1}
I20260927 15:07:40.194909 12252 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5e22578858466c979b11656029bf48 (DEFAULT_TABLE table=test-table [id=c7d72a72acc94380b0d733eaba029f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:40.195576 12121 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5e22578858466c979b11656029bf48 (DEFAULT_TABLE table=test-table [id=c7d72a72acc94380b0d733eaba029f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:40.195576 11990 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5e22578858466c979b11656029bf48 (DEFAULT_TABLE table=test-table [id=c7d72a72acc94380b0d733eaba029f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:40.196184 12252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5e22578858466c979b11656029bf48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:40.196599 12121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5e22578858466c979b11656029bf48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:40.196599 11990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5e22578858466c979b11656029bf48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:40.202589 12335 tablet_bootstrap.cc:492] T 9c5e22578858466c979b11656029bf48 P d66badaf0f0446bcb8aba0ad8bad2270: Bootstrap starting.
I20260927 15:07:40.204715 12337 tablet_bootstrap.cc:492] T 9c5e22578858466c979b11656029bf48 P ae2f44963134453fa55077d73fe9ae9d: Bootstrap starting.
I20260927 15:07:40.205782 12335 tablet_bootstrap.cc:654] T 9c5e22578858466c979b11656029bf48 P d66badaf0f0446bcb8aba0ad8bad2270: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:40.206540 12335 log.cc:824] T 9c5e22578858466c979b11656029bf48 P d66badaf0f0446bcb8aba0ad8bad2270: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:40.206575 12337 tablet_bootstrap.cc:654] T 9c5e22578858466c979b11656029bf48 P ae2f44963134453fa55077d73fe9ae9d: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:40.207201 12337 log.cc:824] T 9c5e22578858466c979b11656029bf48 P ae2f44963134453fa55077d73fe9ae9d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:40.207854 12336 tablet_bootstrap.cc:492] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328: Bootstrap starting.
I20260927 15:07:40.208418 12335 tablet_bootstrap.cc:492] T 9c5e22578858466c979b11656029bf48 P d66badaf0f0446bcb8aba0ad8bad2270: No bootstrap required, opened a new log
I20260927 15:07:40.208631 12335 ts_tablet_manager.cc:1397] T 9c5e22578858466c979b11656029bf48 P d66badaf0f0446bcb8aba0ad8bad2270: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:07:40.209134 12337 tablet_bootstrap.cc:492] T 9c5e22578858466c979b11656029bf48 P ae2f44963134453fa55077d73fe9ae9d: No bootstrap required, opened a new log
I20260927 15:07:40.209316 12337 ts_tablet_manager.cc:1397] T 9c5e22578858466c979b11656029bf48 P ae2f44963134453fa55077d73fe9ae9d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:07:40.209745 12336 tablet_bootstrap.cc:654] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:40.210502 12336 log.cc:824] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:40.212144 12336 tablet_bootstrap.cc:492] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328: No bootstrap required, opened a new log
I20260927 15:07:40.212321 12336 ts_tablet_manager.cc:1397] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 15:07:40.213270 12335 raft_consensus.cc:348] T 9c5e22578858466c979b11656029bf48 P d66badaf0f0446bcb8aba0ad8bad2270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66badaf0f0446bcb8aba0ad8bad2270" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42221 } } peers { permanent_uuid: "12b87f0dd1b2497e8dfd78d7c9b2e328" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33147 } } peers { permanent_uuid: "ae2f44963134453fa55077d73fe9ae9d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 38125 } }
I20260927 15:07:40.213546 12335 raft_consensus.cc:374] T 9c5e22578858466c979b11656029bf48 P d66badaf0f0446bcb8aba0ad8bad2270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:40.213639 12335 raft_consensus.cc:729] T 9c5e22578858466c979b11656029bf48 P d66badaf0f0446bcb8aba0ad8bad2270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66badaf0f0446bcb8aba0ad8bad2270, State: Initialized, Role: FOLLOWER
I20260927 15:07:40.213554 12337 raft_consensus.cc:348] T 9c5e22578858466c979b11656029bf48 P ae2f44963134453fa55077d73fe9ae9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66badaf0f0446bcb8aba0ad8bad2270" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42221 } } peers { permanent_uuid: "12b87f0dd1b2497e8dfd78d7c9b2e328" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33147 } } peers { permanent_uuid: "ae2f44963134453fa55077d73fe9ae9d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 38125 } }
I20260927 15:07:40.213789 12337 raft_consensus.cc:374] T 9c5e22578858466c979b11656029bf48 P ae2f44963134453fa55077d73fe9ae9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:40.213848 12337 raft_consensus.cc:729] T 9c5e22578858466c979b11656029bf48 P ae2f44963134453fa55077d73fe9ae9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae2f44963134453fa55077d73fe9ae9d, State: Initialized, Role: FOLLOWER
I20260927 15:07:40.214124 12335 consensus_queue.cc:260] T 9c5e22578858466c979b11656029bf48 P d66badaf0f0446bcb8aba0ad8bad2270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66badaf0f0446bcb8aba0ad8bad2270" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42221 } } peers { permanent_uuid: "12b87f0dd1b2497e8dfd78d7c9b2e328" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33147 } } peers { permanent_uuid: "ae2f44963134453fa55077d73fe9ae9d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 38125 } }
I20260927 15:07:40.214196 12337 consensus_queue.cc:260] T 9c5e22578858466c979b11656029bf48 P ae2f44963134453fa55077d73fe9ae9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66badaf0f0446bcb8aba0ad8bad2270" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42221 } } peers { permanent_uuid: "12b87f0dd1b2497e8dfd78d7c9b2e328" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33147 } } peers { permanent_uuid: "ae2f44963134453fa55077d73fe9ae9d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 38125 } }
I20260927 15:07:40.214859 12318 heartbeater.cc:503] Master 127.7.17.254:42887 was elected leader, sending a full tablet report...
I20260927 15:07:40.215154 12335 ts_tablet_manager.cc:1428] T 9c5e22578858466c979b11656029bf48 P d66badaf0f0446bcb8aba0ad8bad2270: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260927 15:07:40.215154 12337 ts_tablet_manager.cc:1428] T 9c5e22578858466c979b11656029bf48 P ae2f44963134453fa55077d73fe9ae9d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:07:40.215498 12187 heartbeater.cc:503] Master 127.7.17.254:42887 was elected leader, sending a full tablet report...
I20260927 15:07:40.216259 12336 raft_consensus.cc:348] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66badaf0f0446bcb8aba0ad8bad2270" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42221 } } peers { permanent_uuid: "12b87f0dd1b2497e8dfd78d7c9b2e328" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33147 } } peers { permanent_uuid: "ae2f44963134453fa55077d73fe9ae9d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 38125 } }
I20260927 15:07:40.216538 12336 raft_consensus.cc:374] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:40.216595 12336 raft_consensus.cc:729] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12b87f0dd1b2497e8dfd78d7c9b2e328, State: Initialized, Role: FOLLOWER
I20260927 15:07:40.217053 12336 consensus_queue.cc:260] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66badaf0f0446bcb8aba0ad8bad2270" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42221 } } peers { permanent_uuid: "12b87f0dd1b2497e8dfd78d7c9b2e328" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33147 } } peers { permanent_uuid: "ae2f44963134453fa55077d73fe9ae9d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 38125 } }
I20260927 15:07:40.217689 12056 heartbeater.cc:503] Master 127.7.17.254:42887 was elected leader, sending a full tablet report...
I20260927 15:07:40.218143 12336 ts_tablet_manager.cc:1428] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260927 15:07:40.239735 12188 tablet.cc:2369] T 9c5e22578858466c979b11656029bf48 P ae2f44963134453fa55077d73fe9ae9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:07:40.259461 12343 raft_consensus.cc:482] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:40.259703 12343 raft_consensus.cc:504] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66badaf0f0446bcb8aba0ad8bad2270" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42221 } } peers { permanent_uuid: "12b87f0dd1b2497e8dfd78d7c9b2e328" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33147 } } peers { permanent_uuid: "ae2f44963134453fa55077d73fe9ae9d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 38125 } }
I20260927 15:07:40.260944 12343 leader_election.cc:288] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d66badaf0f0446bcb8aba0ad8bad2270 (127.7.17.195:42221), ae2f44963134453fa55077d73fe9ae9d (127.7.17.194:38125)
I20260927 15:07:40.262876 12333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:40.261315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:51645 (local address 127.7.17.194:38125)
0927 15:07:40.261441 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:40.261446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:40.261462 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:07:40.261590 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:40.261594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:40.261634 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:40.261713 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:40.262510 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:40.262513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:40.262515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:40.262526 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:40.262544 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:40.262552 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:40.262631 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:40.262707 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 15:07:40.262744 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:07:40.262887 12058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:40.261199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:38125 (local address 127.7.17.193:51645)
0927 15:07:40.261335 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:40.261352 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:40.261465 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:40.261749 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:40.261752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:40.261767 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:40.262333 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:07:40.262335 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:40.262346 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:07:40.262374 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:40.262621 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:40.262623 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:40.262626 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 15:07:40.262700 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 15:07:40.262737 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 15:07:40.262954 12330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:40.261418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:42299 (local address 127.7.17.195:42221)
0927 15:07:40.261546 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:40.261551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:40.261798 (+ 247us) server_negotiation.cc:407] Connection header received
0927 15:07:40.261862 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:40.261865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:40.261913 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:40.261998 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:40.262606 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:40.262609 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:40.262612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:40.262624 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:40.262641 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:40.262649 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:40.262728 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:40.262792 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 15:07:40.262829 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:07:40.262998 12344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:40.261199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:42221 (local address 127.7.17.193:42299)
0927 15:07:40.261704 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:40.261716 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:40.261799 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:40.262025 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:40.262028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:40.262042 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:40.262455 (+ 413us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:07:40.262457 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:40.262467 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:07:40.262490 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:40.262720 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:40.262723 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:40.262726 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 15:07:40.262797 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 15:07:40.262829 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":130,"thread_start_us":130,"threads_started":1}
I20260927 15:07:40.263550 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5e22578858466c979b11656029bf48" candidate_uuid: "12b87f0dd1b2497e8dfd78d7c9b2e328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae2f44963134453fa55077d73fe9ae9d" is_pre_election: true
I20260927 15:07:40.263657 12272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5e22578858466c979b11656029bf48" candidate_uuid: "12b87f0dd1b2497e8dfd78d7c9b2e328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66badaf0f0446bcb8aba0ad8bad2270" is_pre_election: true
I20260927 15:07:40.263867 12141 raft_consensus.cc:2454] T 9c5e22578858466c979b11656029bf48 P ae2f44963134453fa55077d73fe9ae9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12b87f0dd1b2497e8dfd78d7c9b2e328 in term 0.
I20260927 15:07:40.263923 12272 raft_consensus.cc:2454] T 9c5e22578858466c979b11656029bf48 P d66badaf0f0446bcb8aba0ad8bad2270 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12b87f0dd1b2497e8dfd78d7c9b2e328 in term 0.
I20260927 15:07:40.264375 11944 leader_election.cc:302] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12b87f0dd1b2497e8dfd78d7c9b2e328, ae2f44963134453fa55077d73fe9ae9d, d66badaf0f0446bcb8aba0ad8bad2270; no voters:
I20260927 15:07:40.264676 12343 raft_consensus.cc:2781] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:40.264756 12343 raft_consensus.cc:482] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:40.264801 12343 raft_consensus.cc:3037] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:40.265838 12343 raft_consensus.cc:504] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66badaf0f0446bcb8aba0ad8bad2270" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42221 } } peers { permanent_uuid: "12b87f0dd1b2497e8dfd78d7c9b2e328" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33147 } } peers { permanent_uuid: "ae2f44963134453fa55077d73fe9ae9d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 38125 } }
I20260927 15:07:40.266168 12343 leader_election.cc:288] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328 [CANDIDATE]: Term 1 election: Requested vote from peers d66badaf0f0446bcb8aba0ad8bad2270 (127.7.17.195:42221), ae2f44963134453fa55077d73fe9ae9d (127.7.17.194:38125)
I20260927 15:07:40.266568 12141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5e22578858466c979b11656029bf48" candidate_uuid: "12b87f0dd1b2497e8dfd78d7c9b2e328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae2f44963134453fa55077d73fe9ae9d"
I20260927 15:07:40.266556 12272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5e22578858466c979b11656029bf48" candidate_uuid: "12b87f0dd1b2497e8dfd78d7c9b2e328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66badaf0f0446bcb8aba0ad8bad2270"
I20260927 15:07:40.266695 12141 raft_consensus.cc:3037] T 9c5e22578858466c979b11656029bf48 P ae2f44963134453fa55077d73fe9ae9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:40.266695 12272 raft_consensus.cc:3037] T 9c5e22578858466c979b11656029bf48 P d66badaf0f0446bcb8aba0ad8bad2270 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:40.267988 12272 raft_consensus.cc:2454] T 9c5e22578858466c979b11656029bf48 P d66badaf0f0446bcb8aba0ad8bad2270 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12b87f0dd1b2497e8dfd78d7c9b2e328 in term 1.
I20260927 15:07:40.267988 12141 raft_consensus.cc:2454] T 9c5e22578858466c979b11656029bf48 P ae2f44963134453fa55077d73fe9ae9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12b87f0dd1b2497e8dfd78d7c9b2e328 in term 1.
I20260927 15:07:40.268304 11944 leader_election.cc:302] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12b87f0dd1b2497e8dfd78d7c9b2e328, ae2f44963134453fa55077d73fe9ae9d; no voters:
I20260927 15:07:40.268537 12343 raft_consensus.cc:2781] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:40.268795 12343 raft_consensus.cc:686] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328 [term 1 LEADER]: Becoming Leader. State: Replica: 12b87f0dd1b2497e8dfd78d7c9b2e328, State: Running, Role: LEADER
I20260927 15:07:40.269214 12343 consensus_queue.cc:237] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66badaf0f0446bcb8aba0ad8bad2270" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42221 } } peers { permanent_uuid: "12b87f0dd1b2497e8dfd78d7c9b2e328" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33147 } } peers { permanent_uuid: "ae2f44963134453fa55077d73fe9ae9d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 38125 } }
I20260927 15:07:40.272406 11868 catalog_manager.cc:5495] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12b87f0dd1b2497e8dfd78d7c9b2e328 (127.7.17.193). New cstate: current_term: 1 leader_uuid: "12b87f0dd1b2497e8dfd78d7c9b2e328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66badaf0f0446bcb8aba0ad8bad2270" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12b87f0dd1b2497e8dfd78d7c9b2e328" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 33147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae2f44963134453fa55077d73fe9ae9d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 38125 } health_report { overall_health: UNKNOWN } } }
I20260927 15:07:40.294235 12334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:40.292758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37762 (local address 127.7.17.193:33147)
0927 15:07:40.292906 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:40.292910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:40.292923 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:07:40.292995 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:40.292998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:40.293056 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:40.293126 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:40.293836 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:40.293839 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:40.293841 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:40.293852 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:40.293872 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:40.293880 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:40.293953 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:40.293995 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 15:07:40.294028 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:07:40.300460 12141 raft_consensus.cc:1261] T 9c5e22578858466c979b11656029bf48 P ae2f44963134453fa55077d73fe9ae9d [term 1 FOLLOWER]: Refusing update from remote peer 12b87f0dd1b2497e8dfd78d7c9b2e328: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:40.300460 12272 raft_consensus.cc:1261] T 9c5e22578858466c979b11656029bf48 P d66badaf0f0446bcb8aba0ad8bad2270 [term 1 FOLLOWER]: Refusing update from remote peer 12b87f0dd1b2497e8dfd78d7c9b2e328: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:40.301343 12345 consensus_queue.cc:1035] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae2f44963134453fa55077d73fe9ae9d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 38125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:40.301544 12343 consensus_queue.cc:1035] T 9c5e22578858466c979b11656029bf48 P 12b87f0dd1b2497e8dfd78d7c9b2e328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66badaf0f0446bcb8aba0ad8bad2270" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:40.309847 12350 mvcc.cc:205] Tried to move back new op lower bound from 7333976720582701056 to 7333976720465248256. Current Snapshot: MvccSnapshot[applied={T|T < 7333976720582701056}]
I20260927 15:07:40.310720 12349 mvcc.cc:205] Tried to move back new op lower bound from 7333976720582701056 to 7333976720465248256. Current Snapshot: MvccSnapshot[applied={T|T < 7333976720582701056}]
I20260927 15:07:40.318342 11868 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:40.318526 11868 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:40.321004 11868 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cd1bbe6742049ec8e7a83b8e7948e7c: Sending DeleteTablet for 3 replicas of tablet 9c5e22578858466c979b11656029bf48
I20260927 15:07:40.321844 12252 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c5e22578858466c979b11656029bf48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:40 UTC) from {username='slave'} at 127.0.0.1:57502
I20260927 15:07:40.322237 11990 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c5e22578858466c979b11656029bf48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:40 UTC) from {username='slave'} at 127.0.0.1:37746
I20260927 15:07:40.322412 12358 tablet_replica.cc:330] T 9c5e22578858466c979b11656029bf48 P d66badaf0f0446bcb8aba0ad8bad2270: stopping tablet replica
I20260927 15:07:40.322394 12121 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c5e22578858466c979b11656029bf48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:40 UTC) from {username='slave'} at 127.0.0.1:53650
I20260927 15:07:40.322739 12358 raft_consensus.cc:2229] T 9c5e22578858466c979b11656029bf48 P d66badaf0f0446bcb8aba0ad8bad2270 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:40.323055 12360 tablet_replica.cc:330] T 9c5e22578858466c979b11656029bf48 P ae2f44963134453fa55077d73fe9ae9d: stopping tablet replica
I20260927 15:07:40.323104 12358 raft_consensus.cc:2258] T 9c5e22578858466c979b11656029bf48 P d66badaf0f0446bcb8aba0ad8bad2270 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:40.323441 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 11928
I20260927 15:07:40.324748 12358 ts_tablet_manager.cc:1905] T 9c5e22578858466c979b11656029bf48 P d66badaf0f0446bcb8aba0ad8bad2270: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:40.325449 12360 raft_consensus.cc:2229] T 9c5e22578858466c979b11656029bf48 P ae2f44963134453fa55077d73fe9ae9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:40.325783 12360 raft_consensus.cc:2258] T 9c5e22578858466c979b11656029bf48 P ae2f44963134453fa55077d73fe9ae9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:40.327201 12360 ts_tablet_manager.cc:1905] T 9c5e22578858466c979b11656029bf48 P ae2f44963134453fa55077d73fe9ae9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:40.327864 12358 ts_tablet_manager.cc:1918] T 9c5e22578858466c979b11656029bf48 P d66badaf0f0446bcb8aba0ad8bad2270: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:40.327965 12358 log.cc:1196] T 9c5e22578858466c979b11656029bf48 P d66badaf0f0446bcb8aba0ad8bad2270: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/9c5e22578858466c979b11656029bf48
I20260927 15:07:40.328269 12358 ts_tablet_manager.cc:1939] T 9c5e22578858466c979b11656029bf48 P d66badaf0f0446bcb8aba0ad8bad2270: Deleting consensus metadata
I20260927 15:07:40.329787 11852 catalog_manager.cc:4841] TS d66badaf0f0446bcb8aba0ad8bad2270 (127.7.17.195:42221): tablet 9c5e22578858466c979b11656029bf48 (table test-table [id=c7d72a72acc94380b0d733eaba029f64]) successfully deleted
W20260927 15:07:40.329847 11855 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.17.193:33147 (error 108)
I20260927 15:07:40.330114 12360 ts_tablet_manager.cc:1918] T 9c5e22578858466c979b11656029bf48 P ae2f44963134453fa55077d73fe9ae9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:40.330199 12360 log.cc:1196] T 9c5e22578858466c979b11656029bf48 P ae2f44963134453fa55077d73fe9ae9d: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521636012660-7239-0/minicluster-data/ts-1/wal/wals/9c5e22578858466c979b11656029bf48
I20260927 15:07:40.330492 12360 ts_tablet_manager.cc:1939] T 9c5e22578858466c979b11656029bf48 P ae2f44963134453fa55077d73fe9ae9d: Deleting consensus metadata
I20260927 15:07:40.330663 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 12059
I20260927 15:07:40.330938 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:40.330651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:33147 (local address 127.0.0.1:37776)
0927 15:07:40.330784 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:40.330855 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.193:33147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 15:07:40.331256 11855 catalog_manager.cc:4584] TS 12b87f0dd1b2497e8dfd78d7c9b2e328 (127.7.17.193:33147): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c5e22578858466c979b11656029bf48: Network error: Client connection negotiation failed: client connection to 127.7.17.193:33147: connect: Connection refused (error 111)
I20260927 15:07:40.336501 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 12190
I20260927 15:07:40.336645 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:40.336385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:38125 (local address 127.0.0.1:53654)
0927 15:07:40.336512 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:40.336562 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.194:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 15:07:40.341538 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 11837
2026-09-27T15:07:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (937 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 15:07:40.361357 7239 mini_oidc.cc:152] Starting JWKS server
I20260927 15:07:40.362915 7239 webserver.cc:460] Webserver started at https://127.0.0.1:35961/ using document root <none> and password file <none>
I20260927 15:07:40.363056 7239 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 15:07:40.363286 7239 webserver.cc:460] Webserver started at http://127.0.0.1:37373/ using document root <none> and password file <none>
2026-09-27T15:07:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:40Z Disabled control of system clock
I20260927 15:07:40.371398 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:38429
--webserver_interface=127.7.17.254
--webserver_port=0
--builtin_ntp_servers=127.7.17.212:39601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:38429
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35961/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521636012660-7239-0/testchainca.cert with env {}
W20260927 15:07:40.482769 12367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:40.483050 12367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:40.483089 12367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:40.487190 12367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:40.487246 12367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:40.487262 12367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:40.487277 12367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:40.487296 12367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:07:40.487311 12367 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:07:40.487326 12367 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35961/jwks.json
I20260927 15:07:40.492728 12367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:38429
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:38429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35961/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521636012660-7239-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:40.493772 12367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:40.494748 12367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:40.503489 12373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:40.503481 12372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:40.503481 12375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:40.503584 12367 server_base.cc:1034] running on GCE node
I20260927 15:07:40.504321 12367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:40.505542 12367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:40.506717 12367 hybrid_clock.cc:648] HybridClock initialized: now 1790521660506676 us; error 44 us; skew 500 ppm
I20260927 15:07:40.508535 12367 webserver.cc:460] Webserver started at http://127.7.17.254:43947/ using document root <none> and password file <none>
I20260927 15:07:40.509166 12367 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:40.509236 12367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:40.509435 12367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:40.511471 12367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "f0d6992585f9407782fffb7bfbba4c79"
format_stamp: "Formatted at 2026-09-27 15:07:40 on dist-test-slave-hhpq"
server_key: "0ece498d634d5f3baf4ad544df62574b"
server_key_iv: "09bf6052464445383444600ff4f07fff"
server_key_version: "encryptionkey@0"
I20260927 15:07:40.512025 12367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "f0d6992585f9407782fffb7bfbba4c79"
format_stamp: "Formatted at 2026-09-27 15:07:40 on dist-test-slave-hhpq"
server_key: "0ece498d634d5f3baf4ad544df62574b"
server_key_iv: "09bf6052464445383444600ff4f07fff"
server_key_version: "encryptionkey@0"
I20260927 15:07:40.515681 12367 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 15:07:40.518210 12381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:40.519440 12367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:07:40.519557 12367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "f0d6992585f9407782fffb7bfbba4c79"
format_stamp: "Formatted at 2026-09-27 15:07:40 on dist-test-slave-hhpq"
server_key: "0ece498d634d5f3baf4ad544df62574b"
server_key_iv: "09bf6052464445383444600ff4f07fff"
server_key_version: "encryptionkey@0"
I20260927 15:07:40.519657 12367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:40.551968 12367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:40.552736 12367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:40.552964 12367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:40.559914 12367 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:38429
I20260927 15:07:40.559926 12433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:38429 every 8 connection(s)
I20260927 15:07:40.560933 12367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:40.563947 12434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:40.567461 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 12367
I20260927 15:07:40.567590 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:07:40.567874 7239 external_mini_cluster.cc:1428] Setting key 24e463a7496775118560ff6ef5487d61
I20260927 15:07:40.570268 12434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0d6992585f9407782fffb7bfbba4c79: Bootstrap starting.
I20260927 15:07:40.572491 12434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0d6992585f9407782fffb7bfbba4c79: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:40.573247 12434 log.cc:824] T 00000000000000000000000000000000 P f0d6992585f9407782fffb7bfbba4c79: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:40.575279 12434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0d6992585f9407782fffb7bfbba4c79: No bootstrap required, opened a new log
I20260927 15:07:40.577430 12437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:40.569459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.7.17.254:38429)
0927 15:07:40.570140 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:40.570160 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:40.570199 (+ 39us) server_negotiation.cc:407] Connection header received
0927 15:07:40.570923 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:40.570943 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:40.571373 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:40.571711 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:40.572158 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.573226 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.574283 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.574760 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.574904 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:40.575684 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:40.575689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:40.575703 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:40.576107 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:40.576165 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:40.576184 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:40.576270 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:40.576404 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 15:07:40.576669 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":140,"threads_started":1}
I20260927 15:07:40.579049 12434 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0d6992585f9407782fffb7bfbba4c79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d6992585f9407782fffb7bfbba4c79" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38429 } }
I20260927 15:07:40.579288 12434 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0d6992585f9407782fffb7bfbba4c79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:40.579348 12434 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0d6992585f9407782fffb7bfbba4c79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d6992585f9407782fffb7bfbba4c79, State: Initialized, Role: FOLLOWER
I20260927 15:07:40.579797 12434 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0d6992585f9407782fffb7bfbba4c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d6992585f9407782fffb7bfbba4c79" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38429 } }
I20260927 15:07:40.579963 12434 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0d6992585f9407782fffb7bfbba4c79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:40.580026 12434 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0d6992585f9407782fffb7bfbba4c79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:40.580127 12434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0d6992585f9407782fffb7bfbba4c79 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:40.580509 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:07:40.581168 12434 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0d6992585f9407782fffb7bfbba4c79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d6992585f9407782fffb7bfbba4c79" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38429 } }
I20260927 15:07:40.581528 12434 leader_election.cc:302] T 00000000000000000000000000000000 P f0d6992585f9407782fffb7bfbba4c79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0d6992585f9407782fffb7bfbba4c79; no voters:
I20260927 15:07:40.581892 12434 leader_election.cc:288] T 00000000000000000000000000000000 P f0d6992585f9407782fffb7bfbba4c79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:40.581980 12439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0d6992585f9407782fffb7bfbba4c79 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:40.582183 12439 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0d6992585f9407782fffb7bfbba4c79 [term 1 LEADER]: Becoming Leader. State: Replica: f0d6992585f9407782fffb7bfbba4c79, State: Running, Role: LEADER
I20260927 15:07:40.582494 12439 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0d6992585f9407782fffb7bfbba4c79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d6992585f9407782fffb7bfbba4c79" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38429 } }
I20260927 15:07:40.583052 12434 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0d6992585f9407782fffb7bfbba4c79 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:40.584084 12441 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0d6992585f9407782fffb7bfbba4c79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0d6992585f9407782fffb7bfbba4c79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d6992585f9407782fffb7bfbba4c79" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38429 } } }
I20260927 15:07:40.584209 12441 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0d6992585f9407782fffb7bfbba4c79 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:40.584475 12440 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0d6992585f9407782fffb7bfbba4c79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0d6992585f9407782fffb7bfbba4c79. Latest consensus state: current_term: 1 leader_uuid: "f0d6992585f9407782fffb7bfbba4c79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d6992585f9407782fffb7bfbba4c79" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38429 } } }
I20260927 15:07:40.584576 12440 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0d6992585f9407782fffb7bfbba4c79 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:40.584554 12448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:40.587776 12448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:40.593931 12448 catalog_manager.cc:1347] Generated new cluster ID: 2a8e105b625f4a9a9f17b910c238b9bc
I20260927 15:07:40.594002 12448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:40.613157 12448 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:40.614156 12448 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:40.618992 12448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0d6992585f9407782fffb7bfbba4c79: Generated new TSK 0
I20260927 15:07:40.619761 12448 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 15:07:40.646786 12464 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 15:07:40.646891 12464 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 15:07:40.647511 12437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:40.629839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41580 (local address 127.7.17.254:38429)
0927 15:07:40.629966 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:40.629971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:40.630127 (+ 156us) server_negotiation.cc:407] Connection header received
0927 15:07:40.630348 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:40.630352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:40.630411 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:40.630528 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:07:40.630798 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.631638 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.632538 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.632928 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.633028 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:40.633420 (+ 392us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:07:40.647074 (+ 13654us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35961/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 15:07:40.647368 (+ 294us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41580: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35961/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 15:07:40.647600 12437 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41580: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35961/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 15:07:40.648787 12463 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:40.629710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:38429 (local address 127.0.0.1:41580)
0927 15:07:40.630039 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:40.630053 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:40.630181 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:40.630559 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:40.630561 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:40.630570 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:07:40.630698 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:40.630703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.631774 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.631777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:40.632434 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:40.632443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.633067 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.633071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:40.633154 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:40.633218 (+ 64us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:07:40.647358 (+ 14140us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35961/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 15:07:40.647627 (+ 269us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.17.254:38429: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35961/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":103,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.7.17.254:38429: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35961/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 15:07:40.649665 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 12367
2026-09-27T15:07:40Z chronyd exiting
I20260927 15:07:40.686239 7239 test_util.cc:175] -----------------------------------------------
I20260927 15:07:40.686354 7239 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521636012660-7239-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (326 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 15:07:40.687594 7239 mini_oidc.cc:152] Starting JWKS server
I20260927 15:07:40.688441 7239 webserver.cc:460] Webserver started at https://127.0.0.1:34643/ using document root <none> and password file <none>
I20260927 15:07:40.688524 7239 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 15:07:40.688740 7239 webserver.cc:460] Webserver started at http://127.0.0.1:35043/ using document root <none> and password file <none>
2026-09-27T15:07:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:40Z Disabled control of system clock
I20260927 15:07:40.697260 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:33085
--webserver_interface=127.7.17.254
--webserver_port=0
--builtin_ntp_servers=127.7.17.212:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:33085
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34643/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521636012660-7239-0/testchainca.cert with env {}
W20260927 15:07:40.806965 12470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:40.807250 12470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:40.807291 12470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:40.811357 12470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:40.811427 12470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:40.811441 12470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:40.811457 12470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:40.811471 12470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:07:40.811486 12470 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:07:40.811497 12470 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34643/jwks.json
I20260927 15:07:40.816998 12470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:33085
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:33085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34643/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521636012660-7239-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:40.818039 12470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:40.819013 12470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:40.827514 12475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:40.827594 12470 server_base.cc:1034] running on GCE node
W20260927 15:07:40.827521 12476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:40.827521 12478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:40.828338 12470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:40.829536 12470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:40.830730 12470 hybrid_clock.cc:648] HybridClock initialized: now 1790521660830703 us; error 42 us; skew 500 ppm
I20260927 15:07:40.832630 12470 webserver.cc:460] Webserver started at http://127.7.17.254:45893/ using document root <none> and password file <none>
I20260927 15:07:40.833235 12470 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:40.833304 12470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:40.833498 12470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:40.835518 12470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "2d6aeef56f30442dae17caf845f2407d"
format_stamp: "Formatted at 2026-09-27 15:07:40 on dist-test-slave-hhpq"
server_key: "f75b79b445f89ea62a5e6387fb38160e"
server_key_iv: "419e03e8f7933f3268887e5502e7a19f"
server_key_version: "encryptionkey@0"
I20260927 15:07:40.836069 12470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "2d6aeef56f30442dae17caf845f2407d"
format_stamp: "Formatted at 2026-09-27 15:07:40 on dist-test-slave-hhpq"
server_key: "f75b79b445f89ea62a5e6387fb38160e"
server_key_iv: "419e03e8f7933f3268887e5502e7a19f"
server_key_version: "encryptionkey@0"
I20260927 15:07:40.839897 12470 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:07:40.842499 12484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:40.843799 12470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:07:40.843920 12470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "2d6aeef56f30442dae17caf845f2407d"
format_stamp: "Formatted at 2026-09-27 15:07:40 on dist-test-slave-hhpq"
server_key: "f75b79b445f89ea62a5e6387fb38160e"
server_key_iv: "419e03e8f7933f3268887e5502e7a19f"
server_key_version: "encryptionkey@0"
I20260927 15:07:40.844029 12470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:40.857275 12470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:40.858023 12470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:40.858239 12470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:40.865295 12470 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:33085
I20260927 15:07:40.865317 12536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:33085 every 8 connection(s)
I20260927 15:07:40.866271 12470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:40.869499 12537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:40.873028 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 12470
I20260927 15:07:40.873144 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:07:40.873420 7239 external_mini_cluster.cc:1428] Setting key dd71539e6fd2b48c007449add1123c24
I20260927 15:07:40.876145 12537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d: Bootstrap starting.
I20260927 15:07:40.878652 12537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:40.879582 12537 log.cc:824] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:40.882011 12537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d: No bootstrap required, opened a new log
I20260927 15:07:40.883055 12540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:40.874842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34206 (local address 127.7.17.254:33085)
0927 15:07:40.875508 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:40.875525 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:40.875564 (+ 39us) server_negotiation.cc:407] Connection header received
0927 15:07:40.876248 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:40.876271 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:40.876716 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:40.877061 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:40.877431 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.878409 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.879604 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.880218 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.880382 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:40.881282 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:40.881287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:40.881301 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:40.881708 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:40.881768 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:40.881788 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:40.881863 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:40.882016 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 15:07:40.882298 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":120,"threads_started":1}
I20260927 15:07:40.885876 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:07:40.885794 12537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6aeef56f30442dae17caf845f2407d" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33085 } }
I20260927 15:07:40.886067 12537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:40.886133 12537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d6aeef56f30442dae17caf845f2407d, State: Initialized, Role: FOLLOWER
I20260927 15:07:40.886656 12537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6aeef56f30442dae17caf845f2407d" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33085 } }
I20260927 15:07:40.886847 12537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:40.886901 12542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:40.887027 12542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6aeef56f30442dae17caf845f2407d" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33085 } }
I20260927 15:07:40.887344 12537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:40.887358 12542 leader_election.cc:302] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d6aeef56f30442dae17caf845f2407d; no voters:
I20260927 15:07:40.887451 12537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:40.887740 12542 leader_election.cc:288] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260927 15:07:40.888495 12537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6aeef56f30442dae17caf845f2407d" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33085 } }
I20260927 15:07:40.888614 12537 leader_election.cc:302] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d6aeef56f30442dae17caf845f2407d; no voters:
I20260927 15:07:40.888710 12537 leader_election.cc:288] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:40.888742 12543 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260927 15:07:40.889070 12542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:40.889070 12544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6aeef56f30442dae17caf845f2407d" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33085 } } }
I20260927 15:07:40.889305 12544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:07:40.889322 12542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d [term 1 LEADER]: Becoming Leader. State: Replica: 2d6aeef56f30442dae17caf845f2407d, State: Running, Role: LEADER
I20260927 15:07:40.889597 12542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6aeef56f30442dae17caf845f2407d" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33085 } }
I20260927 15:07:40.889985 12537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:40.891052 12544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d6aeef56f30442dae17caf845f2407d. Latest consensus state: current_term: 1 leader_uuid: "2d6aeef56f30442dae17caf845f2407d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6aeef56f30442dae17caf845f2407d" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33085 } } }
I20260927 15:07:40.891149 12544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:40.891552 12552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:40.894666 12552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:40.900588 12552 catalog_manager.cc:1347] Generated new cluster ID: f0826abd402d41bb84bb4756a0128203
I20260927 15:07:40.900676 12552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:40.914846 12552 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:40.916194 12552 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:40.931100 12552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d6aeef56f30442dae17caf845f2407d: Generated new TSK 0
I20260927 15:07:40.932013 12552 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 15:07:40.966193 12568 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 15:07:40.966344 12568 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 15:07:40.967193 12540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:40.948539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34218 (local address 127.7.17.254:33085)
0927 15:07:40.948682 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:40.948687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:40.948864 (+ 177us) server_negotiation.cc:407] Connection header received
0927 15:07:40.949067 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:40.949070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:40.949242 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:40.949373 (+ 131us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:07:40.949629 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.950484 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.951383 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.951735 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.951831 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:40.952170 (+ 339us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:07:40.966703 (+ 14533us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34643/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 15:07:40.967020 (+ 317us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34218: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34643/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 15:07:40.967274 12540 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34218: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34643/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 15:07:40.967523 12567 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:40.948450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33085 (local address 127.0.0.1:34218)
0927 15:07:40.948754 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:40.948770 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:40.948932 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:40.949400 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:40.949403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:40.949411 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:07:40.949520 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:40.949524 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.950615 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.950618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:40.951276 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:40.951285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:40.951869 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:40.951872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:40.951955 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:40.951982 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:07:40.966940 (+ 14958us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34643/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 15:07:40.967144 (+ 204us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.17.254:33085: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34643/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":123,"threads_started":1}
I20260927 15:07:40.968076 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 12470
2026-09-27T15:07:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (319 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 15:07:41.006842 7239 mini_oidc.cc:152] Starting JWKS server
I20260927 15:07:41.007675 7239 webserver.cc:460] Webserver started at https://127.0.0.1:44429/ using document root <none> and password file <none>
I20260927 15:07:41.007772 7239 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 15:07:41.007985 7239 webserver.cc:460] Webserver started at http://127.0.0.1:46279/ using document root <none> and password file <none>
2026-09-27T15:07:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:41Z Disabled control of system clock
I20260927 15:07:41.016722 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:40351
--webserver_interface=127.7.17.254
--webserver_port=0
--builtin_ntp_servers=127.7.17.212:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:40351
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44429/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 15:07:41.126433 12574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:41.126724 12574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:41.126762 12574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:41.130810 12574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:41.130885 12574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:41.130899 12574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:41.130915 12574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:41.130928 12574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:07:41.130944 12574 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:07:41.130955 12574 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44429/jwks.json
I20260927 15:07:41.135811 12574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:40351
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:40351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44429/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:41.136973 12574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:41.138345 12574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:07:41.147135 12574 server_base.cc:1034] running on GCE node
W20260927 15:07:41.147086 12580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:41.147086 12579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:41.147090 12582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:41.147979 12574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:41.149255 12574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:41.150429 12574 hybrid_clock.cc:648] HybridClock initialized: now 1790521661150413 us; error 33 us; skew 500 ppm
I20260927 15:07:41.152326 12574 webserver.cc:460] Webserver started at http://127.7.17.254:45785/ using document root <none> and password file <none>
I20260927 15:07:41.152917 12574 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:41.152985 12574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:41.153172 12574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:41.155187 12574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "ef505d7353754696ba5639d1a0771a59"
format_stamp: "Formatted at 2026-09-27 15:07:41 on dist-test-slave-hhpq"
server_key: "3cb90ed2cfc398c1e8d284a242c3998f"
server_key_iv: "b792d2b9c99d655fda903f7e339dd23d"
server_key_version: "encryptionkey@0"
I20260927 15:07:41.155714 12574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "ef505d7353754696ba5639d1a0771a59"
format_stamp: "Formatted at 2026-09-27 15:07:41 on dist-test-slave-hhpq"
server_key: "3cb90ed2cfc398c1e8d284a242c3998f"
server_key_iv: "b792d2b9c99d655fda903f7e339dd23d"
server_key_version: "encryptionkey@0"
I20260927 15:07:41.159478 12574 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:07:41.162189 12588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:41.163638 12574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:07:41.163767 12574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "ef505d7353754696ba5639d1a0771a59"
format_stamp: "Formatted at 2026-09-27 15:07:41 on dist-test-slave-hhpq"
server_key: "3cb90ed2cfc398c1e8d284a242c3998f"
server_key_iv: "b792d2b9c99d655fda903f7e339dd23d"
server_key_version: "encryptionkey@0"
I20260927 15:07:41.163878 12574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:41.179212 12574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:41.179947 12574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:41.180183 12574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:41.186990 12574 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:40351
I20260927 15:07:41.187003 12640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:40351 every 8 connection(s)
I20260927 15:07:41.187987 12574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:41.191401 12641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:41.192595 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 12574
I20260927 15:07:41.192725 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:07:41.193043 7239 external_mini_cluster.cc:1428] Setting key 169324f8e5e9b2ebc2f8ae8868e9b3a5
I20260927 15:07:41.197711 12641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef505d7353754696ba5639d1a0771a59: Bootstrap starting.
I20260927 15:07:41.200423 12641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef505d7353754696ba5639d1a0771a59: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:41.201325 12641 log.cc:824] T 00000000000000000000000000000000 P ef505d7353754696ba5639d1a0771a59: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:41.202628 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:41.194435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34332 (local address 127.7.17.254:40351)
0927 15:07:41.195120 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:41.195137 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:41.195175 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:07:41.195851 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:41.195869 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:41.196303 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:41.196595 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:41.196992 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.198034 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.199229 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.199733 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.199860 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:41.200730 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:41.200736 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:41.200751 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:41.201249 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:41.201314 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:41.201337 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:41.201427 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:41.201577 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 15:07:41.201875 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":146,"threads_started":1}
I20260927 15:07:41.203375 12641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef505d7353754696ba5639d1a0771a59: No bootstrap required, opened a new log
I20260927 15:07:41.205449 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:07:41.207372 12641 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef505d7353754696ba5639d1a0771a59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef505d7353754696ba5639d1a0771a59" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 40351 } }
I20260927 15:07:41.207612 12641 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef505d7353754696ba5639d1a0771a59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:41.207659 12641 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef505d7353754696ba5639d1a0771a59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef505d7353754696ba5639d1a0771a59, State: Initialized, Role: FOLLOWER
I20260927 15:07:41.208033 12641 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef505d7353754696ba5639d1a0771a59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef505d7353754696ba5639d1a0771a59" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 40351 } }
I20260927 15:07:41.208148 12641 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef505d7353754696ba5639d1a0771a59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:41.208196 12641 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef505d7353754696ba5639d1a0771a59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:41.208287 12641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef505d7353754696ba5639d1a0771a59 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:41.209256 12641 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef505d7353754696ba5639d1a0771a59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef505d7353754696ba5639d1a0771a59" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 40351 } }
I20260927 15:07:41.209575 12641 leader_election.cc:302] T 00000000000000000000000000000000 P ef505d7353754696ba5639d1a0771a59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef505d7353754696ba5639d1a0771a59; no voters:
I20260927 15:07:41.209847 12641 leader_election.cc:288] T 00000000000000000000000000000000 P ef505d7353754696ba5639d1a0771a59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:41.210002 12646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef505d7353754696ba5639d1a0771a59 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:41.210217 12646 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef505d7353754696ba5639d1a0771a59 [term 1 LEADER]: Becoming Leader. State: Replica: ef505d7353754696ba5639d1a0771a59, State: Running, Role: LEADER
I20260927 15:07:41.210575 12646 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef505d7353754696ba5639d1a0771a59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef505d7353754696ba5639d1a0771a59" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 40351 } }
I20260927 15:07:41.211045 12641 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef505d7353754696ba5639d1a0771a59 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:41.212107 12647 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef505d7353754696ba5639d1a0771a59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef505d7353754696ba5639d1a0771a59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef505d7353754696ba5639d1a0771a59" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 40351 } } }
I20260927 15:07:41.212236 12647 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef505d7353754696ba5639d1a0771a59 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:41.212508 12648 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef505d7353754696ba5639d1a0771a59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef505d7353754696ba5639d1a0771a59. Latest consensus state: current_term: 1 leader_uuid: "ef505d7353754696ba5639d1a0771a59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef505d7353754696ba5639d1a0771a59" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 40351 } } }
I20260927 15:07:41.212617 12648 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef505d7353754696ba5639d1a0771a59 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:41.212625 12655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:41.216084 12655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:41.222070 12655 catalog_manager.cc:1347] Generated new cluster ID: 6531a5f3205843db8f16947b591a8940
I20260927 15:07:41.222165 12655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:41.229791 12655 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:41.230775 12655 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:41.240410 12655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef505d7353754696ba5639d1a0771a59: Generated new TSK 0
I20260927 15:07:41.241271 12655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:07:41.255226 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:41.249914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34338 (local address 127.7.17.254:40351)
0927 15:07:41.250042 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:41.250047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:41.250181 (+ 134us) server_negotiation.cc:407] Connection header received
0927 15:07:41.250342 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:41.250345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:41.250397 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:41.250495 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:07:41.250748 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.251296 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.252534 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.252784 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.252941 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:41.253349 (+ 408us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:07:41.254797 (+ 1448us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44429/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 15:07:41.255069 (+ 272us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34338: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44429/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 15:07:41.255301 12644 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34338: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44429/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 15:07:41.257085 12670 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:41.249783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:40351 (local address 127.0.0.1:34338)
0927 15:07:41.250085 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:41.250101 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:41.250221 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:41.250512 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:41.250515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:41.250523 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:07:41.250651 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:41.250657 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.251402 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.251406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:41.252397 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:41.252408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.252971 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.252975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:41.253065 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:41.253098 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:07:41.254996 (+ 1898us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44429/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 15:07:41.256894 (+ 1898us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.17.254:40351: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44429/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":1633,"thread_start_us":102,"threads_started":1}
I20260927 15:07:41.257622 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 12574
E20260927 15:07:41.266185 12671 webserver.cc:533] Webserver: sslize failed:
E20260927 15:07:41.266276 12671 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T15:07:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (289 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T15:07:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:41Z Disabled control of system clock
I20260927 15:07:41.304788 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:46431
--webserver_interface=127.7.17.254
--webserver_port=0
--builtin_ntp_servers=127.7.17.212:33091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:46431
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521636012660-7239-0/testchainca.cert with env {}
W20260927 15:07:41.414362 12675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:41.414671 12675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:41.414729 12675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:41.418879 12675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:41.418955 12675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:41.418977 12675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:41.418998 12675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:41.419018 12675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:07:41.419049 12675 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:07:41.419076 12675 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 15:07:41.424722 12675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:46431
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:46431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521636012660-7239-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:41.425905 12675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:41.426990 12675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:41.435729 12681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:41.435880 12675 server_base.cc:1034] running on GCE node
W20260927 15:07:41.435811 12683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:41.435729 12680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:41.436635 12675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:41.437819 12675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:41.439006 12675 hybrid_clock.cc:648] HybridClock initialized: now 1790521661438973 us; error 51 us; skew 500 ppm
I20260927 15:07:41.441160 12675 webserver.cc:460] Webserver started at http://127.7.17.254:42749/ using document root <none> and password file <none>
I20260927 15:07:41.441746 12675 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:41.441814 12675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:41.442009 12675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:41.444152 12675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "c97109fb6da34158aacb1aeaf69fa485"
format_stamp: "Formatted at 2026-09-27 15:07:41 on dist-test-slave-hhpq"
server_key: "574d0bef7c7ea596395fc295ad5cd8c6"
server_key_iv: "7da025cf7b980a9309ddb140f4fd84b6"
server_key_version: "encryptionkey@0"
I20260927 15:07:41.444690 12675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "c97109fb6da34158aacb1aeaf69fa485"
format_stamp: "Formatted at 2026-09-27 15:07:41 on dist-test-slave-hhpq"
server_key: "574d0bef7c7ea596395fc295ad5cd8c6"
server_key_iv: "7da025cf7b980a9309ddb140f4fd84b6"
server_key_version: "encryptionkey@0"
I20260927 15:07:41.448408 12675 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:07:41.450932 12689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:41.452332 12675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 15:07:41.452452 12675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "c97109fb6da34158aacb1aeaf69fa485"
format_stamp: "Formatted at 2026-09-27 15:07:41 on dist-test-slave-hhpq"
server_key: "574d0bef7c7ea596395fc295ad5cd8c6"
server_key_iv: "7da025cf7b980a9309ddb140f4fd84b6"
server_key_version: "encryptionkey@0"
I20260927 15:07:41.452567 12675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:41.477821 12675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:41.478626 12675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:41.478814 12675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:41.485661 12675 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:46431
I20260927 15:07:41.485687 12741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:46431 every 8 connection(s)
I20260927 15:07:41.486677 12675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:41.489970 12742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:41.490789 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 12675
I20260927 15:07:41.490898 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:07:41.491163 7239 external_mini_cluster.cc:1428] Setting key 7d6721c556548fbc1375e8bf8776f2ec
I20260927 15:07:41.495975 12742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c97109fb6da34158aacb1aeaf69fa485: Bootstrap starting.
I20260927 15:07:41.498373 12742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c97109fb6da34158aacb1aeaf69fa485: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:41.499096 12742 log.cc:824] T 00000000000000000000000000000000 P c97109fb6da34158aacb1aeaf69fa485: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:41.500427 12745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:41.492507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40126 (local address 127.7.17.254:46431)
0927 15:07:41.493276 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:41.493294 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:41.493331 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:07:41.493975 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:41.493993 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:41.494457 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:41.494786 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:41.495228 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.496245 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.497161 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.497637 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.497785 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:41.498576 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:41.498581 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:41.498595 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:07:41.498983 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:41.499041 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:07:41.499061 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:41.499156 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:41.499289 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 15:07:41.499574 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":145,"threads_started":1}
I20260927 15:07:41.501293 12742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c97109fb6da34158aacb1aeaf69fa485: No bootstrap required, opened a new log
I20260927 15:07:41.503526 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:07:41.505167 12742 raft_consensus.cc:348] T 00000000000000000000000000000000 P c97109fb6da34158aacb1aeaf69fa485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97109fb6da34158aacb1aeaf69fa485" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46431 } }
I20260927 15:07:41.505411 12742 raft_consensus.cc:374] T 00000000000000000000000000000000 P c97109fb6da34158aacb1aeaf69fa485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:41.505460 12742 raft_consensus.cc:729] T 00000000000000000000000000000000 P c97109fb6da34158aacb1aeaf69fa485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97109fb6da34158aacb1aeaf69fa485, State: Initialized, Role: FOLLOWER
I20260927 15:07:41.505910 12742 consensus_queue.cc:260] T 00000000000000000000000000000000 P c97109fb6da34158aacb1aeaf69fa485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97109fb6da34158aacb1aeaf69fa485" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46431 } }
I20260927 15:07:41.506035 12742 raft_consensus.cc:388] T 00000000000000000000000000000000 P c97109fb6da34158aacb1aeaf69fa485 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:41.506078 12742 raft_consensus.cc:482] T 00000000000000000000000000000000 P c97109fb6da34158aacb1aeaf69fa485 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:41.506175 12742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c97109fb6da34158aacb1aeaf69fa485 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:41.507095 12742 raft_consensus.cc:504] T 00000000000000000000000000000000 P c97109fb6da34158aacb1aeaf69fa485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97109fb6da34158aacb1aeaf69fa485" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46431 } }
I20260927 15:07:41.507421 12742 leader_election.cc:302] T 00000000000000000000000000000000 P c97109fb6da34158aacb1aeaf69fa485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c97109fb6da34158aacb1aeaf69fa485; no voters:
I20260927 15:07:41.507757 12742 leader_election.cc:288] T 00000000000000000000000000000000 P c97109fb6da34158aacb1aeaf69fa485 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:41.507859 12747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c97109fb6da34158aacb1aeaf69fa485 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:41.508095 12747 raft_consensus.cc:686] T 00000000000000000000000000000000 P c97109fb6da34158aacb1aeaf69fa485 [term 1 LEADER]: Becoming Leader. State: Replica: c97109fb6da34158aacb1aeaf69fa485, State: Running, Role: LEADER
I20260927 15:07:41.508409 12747 consensus_queue.cc:237] T 00000000000000000000000000000000 P c97109fb6da34158aacb1aeaf69fa485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97109fb6da34158aacb1aeaf69fa485" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46431 } }
I20260927 15:07:41.509058 12742 sys_catalog.cc:564] T 00000000000000000000000000000000 P c97109fb6da34158aacb1aeaf69fa485 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:41.509694 12749 sys_catalog.cc:455] T 00000000000000000000000000000000 P c97109fb6da34158aacb1aeaf69fa485 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c97109fb6da34158aacb1aeaf69fa485. Latest consensus state: current_term: 1 leader_uuid: "c97109fb6da34158aacb1aeaf69fa485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97109fb6da34158aacb1aeaf69fa485" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46431 } } }
I20260927 15:07:41.509827 12749 sys_catalog.cc:458] T 00000000000000000000000000000000 P c97109fb6da34158aacb1aeaf69fa485 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:41.510039 12748 sys_catalog.cc:455] T 00000000000000000000000000000000 P c97109fb6da34158aacb1aeaf69fa485 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c97109fb6da34158aacb1aeaf69fa485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97109fb6da34158aacb1aeaf69fa485" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46431 } } }
I20260927 15:07:41.510229 12748 sys_catalog.cc:458] T 00000000000000000000000000000000 P c97109fb6da34158aacb1aeaf69fa485 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:41.510316 12755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:41.513283 12755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:41.518608 12755 catalog_manager.cc:1347] Generated new cluster ID: cfdf58430caf4f20b7619891500ef544
I20260927 15:07:41.518688 12755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:41.523355 12755 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:41.524544 12755 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:41.536093 12755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c97109fb6da34158aacb1aeaf69fa485: Generated new TSK 0
I20260927 15:07:41.537032 12755 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:07:41.605775 12745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:41.547284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40142 (local address 127.7.17.254:46431)
0927 15:07:41.547438 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:41.547442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:41.547564 (+ 122us) server_negotiation.cc:407] Connection header received
0927 15:07:41.547728 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:41.547732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:41.547801 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:41.547935 (+ 134us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:07:41.548235 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.549144 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.550078 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.550436 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.550548 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:41.550880 (+ 332us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:07:41.605273 (+ 54393us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 15:07:41.605615 (+ 342us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40142: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 15:07:41.605870 12745 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40142: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 15:07:41.605886 12771 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:41.547161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46431 (local address 127.0.0.1:40142)
0927 15:07:41.547467 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:41.547480 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:41.547592 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:41.547968 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:41.547971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:41.547979 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:07:41.548080 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:41.548084 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.549322 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.549326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:41.549964 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:41.549972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:41.550589 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:41.550592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:41.550670 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:41.550697 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:07:41.605530 (+ 54833us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 15:07:41.605719 (+ 189us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.17.254:46431: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":105,"threads_started":1}
I20260927 15:07:41.606701 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 12675
2026-09-27T15:07:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (328 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (124 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (123 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:07:41 dist-test-slave-hhpq krb5kdc[12783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:07:41 dist-test-slave-hhpq krb5kdc[12783](info): set up 2 sockets
Sep 27 15:07:41 dist-test-slave-hhpq krb5kdc[12783](info): commencing operation
krb5kdc: starting...
W20260927 15:07:42.813040 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:07:42 dist-test-slave-hhpq krb5kdc[12783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:07:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:07:42.931277 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:43169
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:43169
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:43.041714 12796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:43.042022 12796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:43.042083 12796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:43.046458 12796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:43.046532 12796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:43.046561 12796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:43.046581 12796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:43.046602 12796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:43.052078 12796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:43169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:43169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:43.053237 12796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:43.054279 12796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:43.062847 12804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:43.062847 12802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:43.062868 12801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:43.062965 12796 server_base.cc:1034] running on GCE node
I20260927 15:07:43.063796 12796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:43.065081 12796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:43.066273 12796 hybrid_clock.cc:648] HybridClock initialized: now 1790521663066258 us; error 59 us; skew 500 ppm
Sep 27 15:07:43 dist-test-slave-hhpq krb5kdc[12783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521663, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:43.069084 12796 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:07:43.070010 12796 webserver.cc:460] Webserver started at http://127.7.17.254:33811/ using document root <none> and password file <none>
I20260927 15:07:43.070549 12796 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:43.070617 12796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:43.070825 12796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:43.072937 12796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "ca1013704a5049e1862909eacf8583b5"
format_stamp: "Formatted at 2026-09-27 15:07:43 on dist-test-slave-hhpq"
server_key: "5d256bee25f594c415c87eab05e737cd"
server_key_iv: "688338ac327ee7eb4c403298211e5e64"
server_key_version: "encryptionkey@0"
I20260927 15:07:43.073491 12796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "ca1013704a5049e1862909eacf8583b5"
format_stamp: "Formatted at 2026-09-27 15:07:43 on dist-test-slave-hhpq"
server_key: "5d256bee25f594c415c87eab05e737cd"
server_key_iv: "688338ac327ee7eb4c403298211e5e64"
server_key_version: "encryptionkey@0"
I20260927 15:07:43.077208 12796 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:07:43.079551 12811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:43.081003 12796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:07:43.081135 12796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "ca1013704a5049e1862909eacf8583b5"
format_stamp: "Formatted at 2026-09-27 15:07:43 on dist-test-slave-hhpq"
server_key: "5d256bee25f594c415c87eab05e737cd"
server_key_iv: "688338ac327ee7eb4c403298211e5e64"
server_key_version: "encryptionkey@0"
I20260927 15:07:43.081250 12796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:43.091181 12796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:43.093876 12796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:43.094051 12796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:43.100977 12796 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:43169
I20260927 15:07:43.101001 12863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:43169 every 8 connection(s)
I20260927 15:07:43.101953 12796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:43.104682 12864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:43.107535 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 12796
I20260927 15:07:43.107636 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:07:43.107885 7239 external_mini_cluster.cc:1428] Setting key 770f41c40fdfbeee3fe254812fcd1de7
I20260927 15:07:43.111037 12864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca1013704a5049e1862909eacf8583b5: Bootstrap starting.
I20260927 15:07:43.113842 12864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca1013704a5049e1862909eacf8583b5: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:43.114656 12864 log.cc:824] T 00000000000000000000000000000000 P ca1013704a5049e1862909eacf8583b5: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:43.116990 12864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca1013704a5049e1862909eacf8583b5: No bootstrap required, opened a new log
Sep 27 15:07:43 dist-test-slave-hhpq krb5kdc[12783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:43.121150 12864 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca1013704a5049e1862909eacf8583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1013704a5049e1862909eacf8583b5" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43169 } }
I20260927 15:07:43.121445 12864 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca1013704a5049e1862909eacf8583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:43.121490 12864 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca1013704a5049e1862909eacf8583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1013704a5049e1862909eacf8583b5, State: Initialized, Role: FOLLOWER
I20260927 15:07:43.122014 12864 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca1013704a5049e1862909eacf8583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1013704a5049e1862909eacf8583b5" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43169 } }
I20260927 15:07:43.122166 12864 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca1013704a5049e1862909eacf8583b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:43.122237 12864 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca1013704a5049e1862909eacf8583b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:43.122334 12864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca1013704a5049e1862909eacf8583b5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:43.123528 12864 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca1013704a5049e1862909eacf8583b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1013704a5049e1862909eacf8583b5" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43169 } }
I20260927 15:07:43.123915 12864 leader_election.cc:302] T 00000000000000000000000000000000 P ca1013704a5049e1862909eacf8583b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca1013704a5049e1862909eacf8583b5; no voters:
I20260927 15:07:43.124253 12864 leader_election.cc:288] T 00000000000000000000000000000000 P ca1013704a5049e1862909eacf8583b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:43.124405 12869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca1013704a5049e1862909eacf8583b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:43.124557 12867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:43.109454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43910 (local address 127.7.17.254:43169)
0927 15:07:43.109959 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:43.109980 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:43.110026 (+ 46us) server_negotiation.cc:407] Connection header received
0927 15:07:43.110864 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:43.110890 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:43.111353 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:43.111745 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:43.112439 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.113918 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.114891 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.115472 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.115641 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:43.118388 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:43.118395 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:43.118414 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:43.118452 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:43.121158 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:43.121663 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:43.121670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:43.121676 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:43.121822 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:43.122130 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:43.122134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:43.122135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:43.122702 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:43.122898 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:43.123177 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 15:07:43.123503 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":160,"threads_started":1}
I20260927 15:07:43.124620 12869 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca1013704a5049e1862909eacf8583b5 [term 1 LEADER]: Becoming Leader. State: Replica: ca1013704a5049e1862909eacf8583b5, State: Running, Role: LEADER
I20260927 15:07:43.125180 12869 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca1013704a5049e1862909eacf8583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1013704a5049e1862909eacf8583b5" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43169 } }
I20260927 15:07:43.125566 12864 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca1013704a5049e1862909eacf8583b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:43.126929 12870 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca1013704a5049e1862909eacf8583b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca1013704a5049e1862909eacf8583b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1013704a5049e1862909eacf8583b5" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43169 } } }
I20260927 15:07:43.126951 12871 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca1013704a5049e1862909eacf8583b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca1013704a5049e1862909eacf8583b5. Latest consensus state: current_term: 1 leader_uuid: "ca1013704a5049e1862909eacf8583b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1013704a5049e1862909eacf8583b5" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43169 } } }
I20260927 15:07:43.127061 12871 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca1013704a5049e1862909eacf8583b5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:43.127522 12881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:43.127061 12870 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca1013704a5049e1862909eacf8583b5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:43.130838 12881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:43.136448 12881 catalog_manager.cc:1347] Generated new cluster ID: 18355d83be3e4159b7263f28c6be8782
I20260927 15:07:43.136538 12881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:43.157480 12881 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:43.158840 12881 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:43.171355 12881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca1013704a5049e1862909eacf8583b5: Generated new TSK 0
I20260927 15:07:43.172262 12881 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
I20260927 15:07:43.207728 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43169
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:43.350256 12890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:43.350546 12890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:43.350590 12890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:43.354749 12890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:43.354823 12890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:43.354885 12890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:07:43.360293 12890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:43.361486 12890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:43.362545 12890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:43.372324 12896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:43.372390 12895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:43.372336 12898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:43.372565 12890 server_base.cc:1034] running on GCE node
I20260927 15:07:43.373019 12890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:43.373695 12890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:43.374887 12890 hybrid_clock.cc:648] HybridClock initialized: now 1790521663374856 us; error 42 us; skew 500 ppm
Sep 27 15:07:43 dist-test-slave-hhpq krb5kdc[12783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521663, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:43.381544 12890 init.cc:377] Logged in from keytab as kudu/127.7.17.193@KRBTEST.COM (short username kudu)
I20260927 15:07:43.382570 12890 webserver.cc:460] Webserver started at http://127.7.17.193:39533/ using document root <none> and password file <none>
I20260927 15:07:43.383127 12890 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:43.383189 12890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:43.383356 12890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:43.385612 12890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "870dca6ff2b8495d913f5e7a5ff8ff86"
format_stamp: "Formatted at 2026-09-27 15:07:43 on dist-test-slave-hhpq"
server_key: "233da753afdbe7d75b335019f8d539fc"
server_key_iv: "696696df930f6f5ace00f5746a6d22bc"
server_key_version: "encryptionkey@0"
I20260927 15:07:43.386147 12890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "870dca6ff2b8495d913f5e7a5ff8ff86"
format_stamp: "Formatted at 2026-09-27 15:07:43 on dist-test-slave-hhpq"
server_key: "233da753afdbe7d75b335019f8d539fc"
server_key_iv: "696696df930f6f5ace00f5746a6d22bc"
server_key_version: "encryptionkey@0"
I20260927 15:07:43.389933 12890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:07:43.392318 12905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:43.393584 12890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:07:43.393699 12890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "870dca6ff2b8495d913f5e7a5ff8ff86"
format_stamp: "Formatted at 2026-09-27 15:07:43 on dist-test-slave-hhpq"
server_key: "233da753afdbe7d75b335019f8d539fc"
server_key_iv: "696696df930f6f5ace00f5746a6d22bc"
server_key_version: "encryptionkey@0"
I20260927 15:07:43.393792 12890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:43.405064 12890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:43.408268 12890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:43.408429 12890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:43.409046 12890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:43.409917 12890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:43.409963 12890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:43.410006 12890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:43.410034 12890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:43.418370 12890 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:42253
I20260927 15:07:43.418411 13018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:42253 every 8 connection(s)
I20260927 15:07:43.419282 12890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:43.425088 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 12890
I20260927 15:07:43.425341 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:07:43.425642 7239 external_mini_cluster.cc:1428] Setting key 09178d7985f1cdfd71197a33d2ff13d6
Sep 27 15:07:43 dist-test-slave-hhpq krb5kdc[12783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521663, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:43.434338 12867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:43.421131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:36873 (local address 127.7.17.254:43169)
0927 15:07:43.421278 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:43.421283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:43.422288 (+ 1005us) server_negotiation.cc:407] Connection header received
0927 15:07:43.423655 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:43.423658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:43.423708 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:43.423785 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:43.425281 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.425952 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.427421 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.427650 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.427740 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:43.430620 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:43.430623 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:43.430625 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:43.430665 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:43.432023 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:43.432650 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:43.432652 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:43.432653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:43.432729 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:43.433151 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:43.433155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:43.433156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:43.433348 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:43.433429 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:43.434095 (+ 666us) server_negotiation.cc:299] Negotiation successful
0927 15:07:43.434211 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.7.17.194@KRBTEST.COM; defaulting to no policy
I20260927 15:07:43.435520 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:43.421553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:43169 (local address 127.7.17.193:36873)
0927 15:07:43.422122 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:43.422169 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:43.423422 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:43.423954 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:43.423962 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:43.424393 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:43.425013 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:43.425037 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.426132 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.426135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:43.427302 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:43.427312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.427798 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.427802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:43.427971 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:43.428585 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:43.428608 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:43.430427 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:43.432148 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:43.432154 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:43.432174 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:43.432540 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:43.432812 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:43.432815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:43.432817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:43.433046 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:43.433455 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:43.433463 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:43.433824 (+ 361us) client_negotiation.cc:769] Sending connection context
0927 15:07:43.434039 (+ 215us) client_negotiation.cc:240] Negotiation successful
0927 15:07:43.434414 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":125,"threads_started":1}
I20260927 15:07:43.436905 13019 heartbeater.cc:347] Connected to a master server at 127.7.17.254:43169
I20260927 15:07:43.437222 13019 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.194@KRBTEST.COM" created.
I20260927 15:07:43.437778 13019 heartbeater.cc:511] Master 127.7.17.254:43169 requested a full tablet report, sending...
I20260927 15:07:43.439518 12828 ts_manager.cc:194] Registered new tserver with Master: 870dca6ff2b8495d913f5e7a5ff8ff86 (127.7.17.193:42253)
I20260927 15:07:43.440790 12828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:36873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
I20260927 15:07:43.452554 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43169
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:43.562692 13024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:43.563001 13024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:43.563062 13024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:43.567330 13024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:43.567416 13024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:43.567502 13024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:07:43.573122 13024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:43.574249 13024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:43.575295 13024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:43.585450 13032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:43.585466 13029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:43.585476 13024 server_base.cc:1034] running on GCE node
W20260927 15:07:43.585466 13030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:43.586220 13024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:43.586900 13024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:43.588071 13024 hybrid_clock.cc:648] HybridClock initialized: now 1790521663588045 us; error 41 us; skew 500 ppm
Sep 27 15:07:43 dist-test-slave-hhpq krb5kdc[12783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521663, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:43.590955 13024 init.cc:377] Logged in from keytab as kudu/127.7.17.194@KRBTEST.COM (short username kudu)
I20260927 15:07:43.591848 13024 webserver.cc:460] Webserver started at http://127.7.17.194:42069/ using document root <none> and password file <none>
I20260927 15:07:43.592383 13024 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:43.592442 13024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:43.592609 13024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:43.594756 13024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "767d3818dc2f41d290d10929321321f5"
format_stamp: "Formatted at 2026-09-27 15:07:43 on dist-test-slave-hhpq"
server_key: "44987680797a33ea9907a454e129431a"
server_key_iv: "a1fdef504597016bce7e2e8bb0cc7ec0"
server_key_version: "encryptionkey@0"
I20260927 15:07:43.595283 13024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "767d3818dc2f41d290d10929321321f5"
format_stamp: "Formatted at 2026-09-27 15:07:43 on dist-test-slave-hhpq"
server_key: "44987680797a33ea9907a454e129431a"
server_key_iv: "a1fdef504597016bce7e2e8bb0cc7ec0"
server_key_version: "encryptionkey@0"
I20260927 15:07:43.599138 13024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:07:43.601804 13039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:43.603127 13024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:07:43.603302 13024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "767d3818dc2f41d290d10929321321f5"
format_stamp: "Formatted at 2026-09-27 15:07:43 on dist-test-slave-hhpq"
server_key: "44987680797a33ea9907a454e129431a"
server_key_iv: "a1fdef504597016bce7e2e8bb0cc7ec0"
server_key_version: "encryptionkey@0"
I20260927 15:07:43.603411 13024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:43.614723 13024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:43.617988 13024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:43.618158 13024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:43.618762 13024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:43.619567 13024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:43.619609 13024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:43.619648 13024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:43.619673 13024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:43.628252 13024 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:34467
I20260927 15:07:43.628290 13152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:34467 every 8 connection(s)
I20260927 15:07:43.629216 13024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
Sep 27 15:07:43 dist-test-slave-hhpq krb5kdc[12783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521663, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:43.638932 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 13024
I20260927 15:07:43.639024 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:07:43.639271 7239 external_mini_cluster.cc:1428] Setting key 6eb25caa535019c0b32d8e7ecb036930
I20260927 15:07:43.642627 12867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:43.630977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:59287 (local address 127.7.17.254:43169)
0927 15:07:43.631094 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:43.631097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:43.632011 (+ 914us) server_negotiation.cc:407] Connection header received
0927 15:07:43.633457 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:43.633461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:43.633516 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:43.633588 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:43.634770 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.635319 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.636449 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.636673 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.636761 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:43.639200 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:43.639204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:43.639206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:43.639232 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:43.640531 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:43.641092 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:43.641096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:43.641097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:43.641178 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:43.641507 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:43.641511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:43.641512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:43.641704 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:43.641788 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:43.642336 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 15:07:43.642467 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 15:07:43.643497 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:43.631424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:43169 (local address 127.7.17.194:59287)
0927 15:07:43.631864 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:43.631908 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:43.633239 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:43.633759 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:43.633767 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:43.634130 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:43.634632 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:43.634646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.635441 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.635444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:43.636332 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:43.636341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.636846 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.636851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:43.637010 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:43.637561 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:43.637577 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:43.638985 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:43.640642 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:43.640648 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:43.640667 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:43.641002 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:43.641258 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:43.641261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:43.641263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:43.641410 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:43.641800 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:43.641807 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:43.642083 (+ 276us) client_negotiation.cc:769] Sending connection context
0927 15:07:43.642262 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 15:07:43.642609 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":116,"threads_started":1}
I20260927 15:07:43.644642 13153 heartbeater.cc:347] Connected to a master server at 127.7.17.254:43169
I20260927 15:07:43.644951 13153 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:43.645562 13153 heartbeater.cc:511] Master 127.7.17.254:43169 requested a full tablet report, sending...
I20260927 15:07:43.646731 12828 ts_manager.cc:194] Registered new tserver with Master: 767d3818dc2f41d290d10929321321f5 (127.7.17.194:34467)
I20260927 15:07:43.647327 12828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:59287
WARNING: no policy specified for kudu/127.7.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
I20260927 15:07:43.667196 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43169
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:43.779277 13158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:43.779582 13158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:43.779624 13158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:43.783756 13158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:43.783824 13158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:43.783887 13158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:07:43.789551 13158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:43.790644 13158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:43.791661 13158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:43.801561 13163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:43.801527 13164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:43.801658 13158 server_base.cc:1034] running on GCE node
W20260927 15:07:43.801558 13166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:43.802377 13158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:43.803037 13158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:43.804210 13158 hybrid_clock.cc:648] HybridClock initialized: now 1790521663804179 us; error 46 us; skew 500 ppm
Sep 27 15:07:43 dist-test-slave-hhpq krb5kdc[12783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521663, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:43.807116 13158 init.cc:377] Logged in from keytab as kudu/127.7.17.195@KRBTEST.COM (short username kudu)
I20260927 15:07:43.808077 13158 webserver.cc:460] Webserver started at http://127.7.17.195:41495/ using document root <none> and password file <none>
I20260927 15:07:43.808607 13158 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:43.808666 13158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:43.808872 13158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:43.810949 13158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "e51c8662b2da4a0e972b982f3c916c98"
format_stamp: "Formatted at 2026-09-27 15:07:43 on dist-test-slave-hhpq"
server_key: "35ccbc27d51d86a0b2abb0a06357ae9d"
server_key_iv: "a26835cecee68e5800779cb036dd3e94"
server_key_version: "encryptionkey@0"
I20260927 15:07:43.811465 13158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "e51c8662b2da4a0e972b982f3c916c98"
format_stamp: "Formatted at 2026-09-27 15:07:43 on dist-test-slave-hhpq"
server_key: "35ccbc27d51d86a0b2abb0a06357ae9d"
server_key_iv: "a26835cecee68e5800779cb036dd3e94"
server_key_version: "encryptionkey@0"
I20260927 15:07:43.815168 13158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:07:43.817660 13173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:43.819137 13158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:43.819255 13158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "e51c8662b2da4a0e972b982f3c916c98"
format_stamp: "Formatted at 2026-09-27 15:07:43 on dist-test-slave-hhpq"
server_key: "35ccbc27d51d86a0b2abb0a06357ae9d"
server_key_iv: "a26835cecee68e5800779cb036dd3e94"
server_key_version: "encryptionkey@0"
I20260927 15:07:43.819380 13158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:43.830193 13158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:43.833516 13158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:43.833698 13158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:43.834298 13158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:43.835146 13158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:43.835203 13158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:43.835243 13158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:43.835270 13158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:43.843704 13158 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:35163
I20260927 15:07:43.843724 13286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:35163 every 8 connection(s)
I20260927 15:07:43.844631 13158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:43.853857 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 13158
I20260927 15:07:43.853989 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:07:43.854301 7239 external_mini_cluster.cc:1428] Setting key 1fe6960dff37ac8a98819a8a497d84b7
Sep 27 15:07:43 dist-test-slave-hhpq krb5kdc[12783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521663, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:43.859172 12867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:43.846583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:58935 (local address 127.7.17.254:43169)
0927 15:07:43.846725 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:43.846730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:43.847718 (+ 988us) server_negotiation.cc:407] Connection header received
0927 15:07:43.849189 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:43.849193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:43.849245 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:43.849324 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:43.850611 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.851149 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.852444 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.852685 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.852779 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:43.855939 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:43.855942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:43.855943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:43.855969 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:43.857377 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:43.857879 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:43.857882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:43.857883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:43.857954 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:43.858221 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:43.858224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:43.858225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:43.858397 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:43.858469 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:43.858961 (+ 492us) server_negotiation.cc:299] Negotiation successful
0927 15:07:43.859054 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:07:43.860040 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:43.846989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:43169 (local address 127.7.17.195:58935)
0927 15:07:43.847556 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:43.847602 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:43.848941 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:43.849486 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:43.849494 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:43.849869 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:43.850435 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:43.850449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.851270 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.851273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:43.852303 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:43.852313 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.852907 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.852912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:43.853093 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:43.853696 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:43.853718 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:43.855667 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:43.857484 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:43.857489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:43.857504 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:43.857801 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:43.858018 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:43.858020 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:43.858021 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:43.858158 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:43.858491 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:43.858497 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:43.858770 (+ 273us) client_negotiation.cc:769] Sending connection context
0927 15:07:43.858932 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 15:07:43.859202 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":131,"threads_started":1}
I20260927 15:07:43.861263 13287 heartbeater.cc:347] Connected to a master server at 127.7.17.254:43169
I20260927 15:07:43.861515 13287 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:43.862032 13287 heartbeater.cc:511] Master 127.7.17.254:43169 requested a full tablet report, sending...
I20260927 15:07:43.863219 12828 ts_manager.cc:194] Registered new tserver with Master: e51c8662b2da4a0e972b982f3c916c98 (127.7.17.195:35163)
I20260927 15:07:43.863770 12828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:58935
I20260927 15:07:43.868587 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/test-admin.keytab.
Sep 27 15:07:43 dist-test-slave-hhpq krb5kdc[12783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:43.886047 7239 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 15:07:43.889093 13294 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:43.887436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36968 (local address 127.7.17.193:42253)
0927 15:07:43.887804 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:43.887815 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:43.887839 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:07:43.887889 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:43.887895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:43.888085 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:43.888225 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:43.888888 (+ 663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36968: BlockingRecv error: recv got EOF from 127.0.0.1:36968 (error 108)
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":151,"threads_started":1}
W20260927 15:07:43.889284 7239 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/krb5cc' of type FILE
Sep 27 15:07:43 dist-test-slave-hhpq krb5kdc[12783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:43.890344 7239 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 15:07:43 dist-test-slave-hhpq krb5kdc[12783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:43.891261 7239 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 15:07:43 dist-test-slave-hhpq krb5kdc[12783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.193@KRBTEST.COM
I20260927 15:07:43.901425 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:43.892238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36978 (local address 127.7.17.193:42253)
0927 15:07:43.892348 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:43.892352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:43.892440 (+ 88us) server_negotiation.cc:407] Connection header received
0927 15:07:43.892546 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:43.892548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:43.892590 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:43.892664 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:43.893303 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.894102 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.894997 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.895396 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.895485 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:43.897888 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:43.897898 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:43.897913 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:43.897942 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:43.899578 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:43.900017 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:43.900023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:43.900024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:43.900105 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:43.900392 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:43.900395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:43.900396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:43.900664 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:43.900905 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:43.901083 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 15:07:43.901228 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 15:07:43.905481 13294 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:43.904364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36986 (local address 127.7.17.193:42253)
0927 15:07:43.904478 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:43.904481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:43.904627 (+ 146us) server_negotiation.cc:407] Connection header received
0927 15:07:43.904732 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:43.904734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:43.904771 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:43.904898 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:43.905368 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36986: BlockingRecv error: recv got EOF from 127.0.0.1:36986 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 15:07:43.905674 7239 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/krb5cc' of type FILE
Sep 27 15:07:43 dist-test-slave-hhpq krb5kdc[12783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:43.906715 7239 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 15:07:43 dist-test-slave-hhpq krb5kdc[12783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:43.907613 7239 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 15:07:43 dist-test-slave-hhpq krb5kdc[12783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.193@KRBTEST.COM
I20260927 15:07:43.916342 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:43.908526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37000 (local address 127.7.17.193:42253)
0927 15:07:43.908679 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:43.908683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:43.908759 (+ 76us) server_negotiation.cc:407] Connection header received
0927 15:07:43.908940 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:43.908943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:43.908983 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:43.909048 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:43.909618 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.910133 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.910888 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.911110 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.911199 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:43.913498 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:43.913502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:43.913504 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:43.913530 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:43.914879 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:43.915267 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:43.915271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:43.915272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:43.915344 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:43.915574 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:43.915577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:43.915579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:43.915756 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:43.915865 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:43.916081 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 15:07:43.916190 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 15:07:43.918947 13294 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:43.917995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37008 (local address 127.7.17.193:42253)
0927 15:07:43.918117 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:43.918120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:43.918248 (+ 128us) server_negotiation.cc:407] Connection header received
0927 15:07:43.918317 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:43.918319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:43.918355 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:43.918415 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:43.918852 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37008: BlockingRecv error: recv got EOF from 127.0.0.1:37008 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 15:07:43.918982 7239 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521636012660-7239-0/krb5kdc/krb5cc' of type FILE
Sep 27 15:07:43 dist-test-slave-hhpq krb5kdc[12783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:43.920019 7239 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 15:07:43 dist-test-slave-hhpq krb5kdc[12783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:43.920980 7239 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 15:07:43 dist-test-slave-hhpq krb5kdc[12783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.193@KRBTEST.COM
I20260927 15:07:43.929864 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:43.921881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37022 (local address 127.7.17.193:42253)
0927 15:07:43.922034 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:43.922038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:43.922162 (+ 124us) server_negotiation.cc:407] Connection header received
0927 15:07:43.922272 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:43.922275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:43.922318 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:43.922381 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:43.922880 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.923419 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.924241 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:43.924447 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:43.924529 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:43.926931 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:43.926934 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:43.926936 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:43.926959 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:43.928339 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:43.928725 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:43.928727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:43.928729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:43.928806 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:43.929077 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:43.929080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:43.929081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:43.929295 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:43.929392 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:43.929642 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 15:07:43.929744 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:07:43.930785 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 12890
I20260927 15:07:43.935685 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 13024
I20260927 15:07:43.940112 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 13158
I20260927 15:07:43.945013 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 12796
2026-09-27T15:07:43Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2092 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:07:44 dist-test-slave-hhpq krb5kdc[13307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:07:44 dist-test-slave-hhpq krb5kdc[13307](info): set up 2 sockets
Sep 27 15:07:44 dist-test-slave-hhpq krb5kdc[13307](info): commencing operation
krb5kdc: starting...
W20260927 15:07:44.908973 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:07:44 dist-test-slave-hhpq krb5kdc[13307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:07:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.254.keytab.
Entry for principal oryx/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.254.keytab.
I20260927 15:07:45.031744 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:33177
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.254.keytab
--principal=oryx/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:33177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:45.142411 13320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:45.142710 13320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:45.142757 13320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:45.146912 13320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:45.146978 13320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:45.146996 13320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:45.147011 13320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:45.147025 13320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:45.152567 13320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:33177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.254.keytab
--principal=oryx/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:33177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:45.153685 13320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:45.154716 13320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:07:45.163435 13320 server_base.cc:1034] running on GCE node
W20260927 15:07:45.163374 13328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:45.163388 13325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:45.163383 13326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:45.164428 13320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:45.165755 13320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:45.166953 13320 hybrid_clock.cc:648] HybridClock initialized: now 1790521665166933 us; error 42 us; skew 500 ppm
Sep 27 15:07:45 dist-test-slave-hhpq krb5kdc[13307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521665, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:45.170039 13320 init.cc:377] Logged in from keytab as oryx/127.7.17.254@KRBTEST.COM (short username oryx)
I20260927 15:07:45.171010 13320 webserver.cc:460] Webserver started at http://127.7.17.254:42303/ using document root <none> and password file <none>
I20260927 15:07:45.171595 13320 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:45.171666 13320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:45.171877 13320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:45.174117 13320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "bc7b3a82e94e47b28a0f16ceb4f31721"
format_stamp: "Formatted at 2026-09-27 15:07:45 on dist-test-slave-hhpq"
server_key: "dcbd97e0b5b49d1e42c6da44ca45a4b1"
server_key_iv: "2dc5bbf00c68c6ad4934939a4fe054e3"
server_key_version: "encryptionkey@0"
I20260927 15:07:45.174698 13320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "bc7b3a82e94e47b28a0f16ceb4f31721"
format_stamp: "Formatted at 2026-09-27 15:07:45 on dist-test-slave-hhpq"
server_key: "dcbd97e0b5b49d1e42c6da44ca45a4b1"
server_key_iv: "2dc5bbf00c68c6ad4934939a4fe054e3"
server_key_version: "encryptionkey@0"
I20260927 15:07:45.178715 13320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:07:45.181502 13335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:45.182964 13320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:45.183109 13320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "bc7b3a82e94e47b28a0f16ceb4f31721"
format_stamp: "Formatted at 2026-09-27 15:07:45 on dist-test-slave-hhpq"
server_key: "dcbd97e0b5b49d1e42c6da44ca45a4b1"
server_key_iv: "2dc5bbf00c68c6ad4934939a4fe054e3"
server_key_version: "encryptionkey@0"
I20260927 15:07:45.183236 13320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:45.203676 13320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:45.206984 13320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:45.207176 13320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:45.214165 13320 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:33177
I20260927 15:07:45.214421 13387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:33177 every 8 connection(s)
I20260927 15:07:45.215149 13320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:45.218051 13388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:45.218137 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 13320
I20260927 15:07:45.218277 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:07:45.218583 7239 external_mini_cluster.cc:1428] Setting key f697bdca9f9eb73468ecf06ee06f8e9b
I20260927 15:07:45.224635 13388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721: Bootstrap starting.
I20260927 15:07:45.227345 13388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721: Neither blocks nor log segments found. Creating new log.
Sep 27 15:07:45 dist-test-slave-hhpq krb5kdc[13307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM
I20260927 15:07:45.228113 13388 log.cc:824] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:45.230759 13388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721: No bootstrap required, opened a new log
I20260927 15:07:45.234182 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:45.220163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44186 (local address 127.7.17.254:33177)
0927 15:07:45.220623 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:45.220638 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:45.220675 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:07:45.221502 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:45.221524 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:45.221875 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:45.222212 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:45.223005 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.223993 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.224893 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.225389 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.225507 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:45.228414 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:45.228422 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:45.228444 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:45.228479 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:45.230671 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:45.231214 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:45.231220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:45.231227 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:45.231380 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:45.231669 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:45.231672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:45.231674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:45.232240 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:45.232433 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:45.232732 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 15:07:45.233103 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":132,"threads_started":1}
I20260927 15:07:45.235445 13388 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } }
I20260927 15:07:45.235719 13388 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:45.235759 13388 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc7b3a82e94e47b28a0f16ceb4f31721, State: Initialized, Role: FOLLOWER
I20260927 15:07:45.236235 13388 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } }
I20260927 15:07:45.236364 13388 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:45.236424 13388 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:45.236526 13388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:45.237545 13388 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } }
I20260927 15:07:45.237974 13388 leader_election.cc:302] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc7b3a82e94e47b28a0f16ceb4f31721; no voters:
I20260927 15:07:45.238320 13388 leader_election.cc:288] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:45.238493 13393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:45.238739 13393 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [term 1 LEADER]: Becoming Leader. State: Replica: bc7b3a82e94e47b28a0f16ceb4f31721, State: Running, Role: LEADER
I20260927 15:07:45.239118 13393 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } }
I20260927 15:07:45.239698 13388 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:45.240463 13395 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } }
I20260927 15:07:45.240597 13395 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:45.241282 13394 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc7b3a82e94e47b28a0f16ceb4f31721. Latest consensus state: current_term: 1 leader_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } }
I20260927 15:07:45.241370 13394 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:45.241850 13402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:45.245196 13402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:45.250686 13402 catalog_manager.cc:1347] Generated new cluster ID: 79c4dfdb3ecf46408724e71fcbeb7be8
I20260927 15:07:45.250753 13402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:45.259975 13402 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:45.261054 13402 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:45.271483 13402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721: Generated new TSK 0
I20260927 15:07:45.272454 13402 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.193.keytab.
Entry for principal oryx/127.7.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.193.keytab.
I20260927 15:07:45.312171 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:33177
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.193.keytab
--principal=oryx/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:45.424755 13414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:45.425074 13414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:45.425136 13414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:45.429365 13414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:45.429445 13414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:45.429527 13414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:07:45.435005 13414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.193.keytab
--principal=oryx/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:33177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:45.436142 13414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:45.437251 13414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:07:45.447489 13414 server_base.cc:1034] running on GCE node
W20260927 15:07:45.447379 13420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:45.447399 13419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:45.447378 13422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:45.448163 13414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:45.448952 13414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:45.450117 13414 hybrid_clock.cc:648] HybridClock initialized: now 1790521665450072 us; error 61 us; skew 500 ppm
Sep 27 15:07:45 dist-test-slave-hhpq krb5kdc[13307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521665, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:45.452932 13414 init.cc:377] Logged in from keytab as oryx/127.7.17.193@KRBTEST.COM (short username oryx)
I20260927 15:07:45.453902 13414 webserver.cc:460] Webserver started at http://127.7.17.193:44421/ using document root <none> and password file <none>
I20260927 15:07:45.454473 13414 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:45.454543 13414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:45.454733 13414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:45.456879 13414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "627b253d79314dd39ba841bfedb233e6"
format_stamp: "Formatted at 2026-09-27 15:07:45 on dist-test-slave-hhpq"
server_key: "e4c9a1f7c5d5f2f7d4f337d5c47bce2f"
server_key_iv: "22eead3a68f39c61cfec8b4fd58eeb66"
server_key_version: "encryptionkey@0"
I20260927 15:07:45.457450 13414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "627b253d79314dd39ba841bfedb233e6"
format_stamp: "Formatted at 2026-09-27 15:07:45 on dist-test-slave-hhpq"
server_key: "e4c9a1f7c5d5f2f7d4f337d5c47bce2f"
server_key_iv: "22eead3a68f39c61cfec8b4fd58eeb66"
server_key_version: "encryptionkey@0"
I20260927 15:07:45.461491 13414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:07:45.463975 13429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:45.465497 13414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:45.465623 13414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "627b253d79314dd39ba841bfedb233e6"
format_stamp: "Formatted at 2026-09-27 15:07:45 on dist-test-slave-hhpq"
server_key: "e4c9a1f7c5d5f2f7d4f337d5c47bce2f"
server_key_iv: "22eead3a68f39c61cfec8b4fd58eeb66"
server_key_version: "encryptionkey@0"
I20260927 15:07:45.465744 13414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:45.484243 13414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:45.487525 13414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:45.487705 13414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:45.488293 13414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:45.489177 13414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:45.489233 13414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:45.489285 13414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:45.489310 13414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:45.497609 13414 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:45549
I20260927 15:07:45.497648 13542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:45549 every 8 connection(s)
I20260927 15:07:45.498536 13414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:45.498917 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 13414
I20260927 15:07:45.499042 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:07:45.499352 7239 external_mini_cluster.cc:1428] Setting key cee38bddefffd8ddfed91dffee51e405
WARNING: no policy specified for oryx/127.7.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.17.194@KRBTEST.COM" created.
Sep 27 15:07:45 dist-test-slave-hhpq krb5kdc[13307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521665, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM
I20260927 15:07:45.516680 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:45.500397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:48591 (local address 127.7.17.254:33177)
0927 15:07:45.500526 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:45.500530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:45.502048 (+ 1518us) server_negotiation.cc:407] Connection header received
0927 15:07:45.503052 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:45.503141 (+ 89us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:45.503202 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:45.503303 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:45.504923 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.505571 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.507292 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.507533 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.507638 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:45.512684 (+ 5046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:45.512688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:45.512691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:45.512724 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:45.514309 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:45.514919 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:45.514922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:45.514923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:45.515011 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:45.515435 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:45.515438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:45.515439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:45.515662 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:45.515759 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:45.516349 (+ 590us) server_negotiation.cc:299] Negotiation successful
0927 15:07:45.516495 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:07:45.517585 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:45.500931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.193:48591)
0927 15:07:45.501872 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:45.501935 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:45.502850 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:45.503540 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:45.503549 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:45.504039 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:45.504723 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:45.504745 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.505712 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.505716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:45.507149 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:45.507161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.507699 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.507703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:45.507908 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:45.508541 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:45.508563 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:45.512465 (+ 3902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:45.514453 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:45.514459 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:45.514474 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:45.514791 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:45.515154 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:45.515157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:45.515159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:45.515320 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:45.515777 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:45.515784 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:45.516071 (+ 287us) client_negotiation.cc:769] Sending connection context
0927 15:07:45.516267 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 15:07:45.516645 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":522,"threads_started":1}
I20260927 15:07:45.519104 13543 heartbeater.cc:347] Connected to a master server at 127.7.17.254:33177
I20260927 15:07:45.519366 13543 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:45.519898 13543 heartbeater.cc:511] Master 127.7.17.254:33177 requested a full tablet report, sending...
I20260927 15:07:45.521652 13351 ts_manager.cc:194] Registered new tserver with Master: 627b253d79314dd39ba841bfedb233e6 (127.7.17.193:45549)
I20260927 15:07:45.522940 13351 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:48591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.194.keytab.
Entry for principal oryx/127.7.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.194.keytab.
I20260927 15:07:45.528234 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:33177
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.194.keytab
--principal=oryx/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:45.639895 13548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:45.640173 13548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:45.640231 13548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:45.644399 13548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:45.644460 13548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:45.644531 13548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:07:45.649967 13548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.194.keytab
--principal=oryx/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:33177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:45.651026 13548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:45.652025 13548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:45.661586 13553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:45.661587 13554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:45.661713 13548 server_base.cc:1034] running on GCE node
W20260927 15:07:45.661586 13556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:45.662351 13548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:45.663018 13548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:45.664216 13548 hybrid_clock.cc:648] HybridClock initialized: now 1790521665664192 us; error 38 us; skew 500 ppm
Sep 27 15:07:45 dist-test-slave-hhpq krb5kdc[13307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521665, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:45.667029 13548 init.cc:377] Logged in from keytab as oryx/127.7.17.194@KRBTEST.COM (short username oryx)
I20260927 15:07:45.668026 13548 webserver.cc:460] Webserver started at http://127.7.17.194:41513/ using document root <none> and password file <none>
I20260927 15:07:45.668581 13548 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:45.668654 13548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:45.668895 13548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:45.671015 13548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d"
format_stamp: "Formatted at 2026-09-27 15:07:45 on dist-test-slave-hhpq"
server_key: "e68592eb19f7a984159aff53125047aa"
server_key_iv: "e33edcff548ea33c41ce61a4e5d5273b"
server_key_version: "encryptionkey@0"
I20260927 15:07:45.671561 13548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d"
format_stamp: "Formatted at 2026-09-27 15:07:45 on dist-test-slave-hhpq"
server_key: "e68592eb19f7a984159aff53125047aa"
server_key_iv: "e33edcff548ea33c41ce61a4e5d5273b"
server_key_version: "encryptionkey@0"
I20260927 15:07:45.675549 13548 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 15:07:45.678198 13563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:45.679620 13548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:07:45.679771 13548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d"
format_stamp: "Formatted at 2026-09-27 15:07:45 on dist-test-slave-hhpq"
server_key: "e68592eb19f7a984159aff53125047aa"
server_key_iv: "e33edcff548ea33c41ce61a4e5d5273b"
server_key_version: "encryptionkey@0"
I20260927 15:07:45.679879 13548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:45.700294 13548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:45.703397 13548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:45.703593 13548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:45.704188 13548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:45.705124 13548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:45.705173 13548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:45.705224 13548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:45.705253 13548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:45.713666 13548 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:46871
I20260927 15:07:45.713740 13676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:46871 every 8 connection(s)
I20260927 15:07:45.714582 13548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:07:45.714910 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 13548
I20260927 15:07:45.715054 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:07:45.715376 7239 external_mini_cluster.cc:1428] Setting key ccafb8c133dd83ae3fb0d579387a6d80
WARNING: no policy specified for oryx/127.7.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.17.195@KRBTEST.COM" created.
Sep 27 15:07:45 dist-test-slave-hhpq krb5kdc[13307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521665, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM
I20260927 15:07:45.731475 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:45.716543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:49585 (local address 127.7.17.254:33177)
0927 15:07:45.716663 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:45.716668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:45.718045 (+ 1377us) server_negotiation.cc:407] Connection header received
0927 15:07:45.719116 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:45.719119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:45.719176 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:45.719247 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:45.720645 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.721219 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.722397 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.722630 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.722722 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:45.727799 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:45.727803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:45.727805 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:45.727839 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:45.729314 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:45.729851 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:45.729854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:45.729856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:45.729947 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:45.730260 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:45.730264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:45.730266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:45.730505 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:45.730623 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:45.731150 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 15:07:45.731295 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 15:07:45.732373 13679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:45.717085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.194:49585)
0927 15:07:45.717866 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:45.717927 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:45.718897 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:45.719439 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:45.719446 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:45.719823 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:45.720464 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:45.720480 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.721347 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.721350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:45.722274 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:45.722284 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.722782 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.722786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:45.722957 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:45.723555 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:45.723572 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:45.727552 (+ 3980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:45.729453 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:45.729459 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:45.729477 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:45.729754 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:45.730025 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:45.730027 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:45.730029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:45.730177 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:45.730625 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:45.730631 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:45.730895 (+ 264us) client_negotiation.cc:769] Sending connection context
0927 15:07:45.731068 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 15:07:45.731405 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":115,"threads_started":1}
I20260927 15:07:45.733969 13677 heartbeater.cc:347] Connected to a master server at 127.7.17.254:33177
I20260927 15:07:45.734216 13677 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:45.734740 13677 heartbeater.cc:511] Master 127.7.17.254:33177 requested a full tablet report, sending...
I20260927 15:07:45.735831 13351 ts_manager.cc:194] Registered new tserver with Master: 3a3d3d46c72546e48fdcdcc8b8996d6d (127.7.17.194:46871)
I20260927 15:07:45.736485 13351 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:49585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.195.keytab.
Entry for principal oryx/127.7.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.195.keytab.
I20260927 15:07:45.743242 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:33177
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.195.keytab
--principal=oryx/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:45.855122 13682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:45.855415 13682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:45.855463 13682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:45.859522 13682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:45.859586 13682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:45.859644 13682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:07:45.865013 13682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.195.keytab
--principal=oryx/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:33177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:45.866060 13682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:45.867070 13682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:45.876672 13690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:45.876768 13688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:45.876685 13687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:45.876885 13682 server_base.cc:1034] running on GCE node
I20260927 15:07:45.877585 13682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:45.878356 13682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:45.879534 13682 hybrid_clock.cc:648] HybridClock initialized: now 1790521665879516 us; error 42 us; skew 500 ppm
Sep 27 15:07:45 dist-test-slave-hhpq krb5kdc[13307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521665, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:45.882500 13682 init.cc:377] Logged in from keytab as oryx/127.7.17.195@KRBTEST.COM (short username oryx)
I20260927 15:07:45.883580 13682 webserver.cc:460] Webserver started at http://127.7.17.195:37761/ using document root <none> and password file <none>
I20260927 15:07:45.884209 13682 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:45.884287 13682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:45.884500 13682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:45.886828 13682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "be770c2746a2448e9afa82fdf76f08ec"
format_stamp: "Formatted at 2026-09-27 15:07:45 on dist-test-slave-hhpq"
server_key: "89c5e459ed11afabb49e73424e88d84c"
server_key_iv: "a8f4288850baf8c11c8b7f641b5ecf11"
server_key_version: "encryptionkey@0"
I20260927 15:07:45.887442 13682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "be770c2746a2448e9afa82fdf76f08ec"
format_stamp: "Formatted at 2026-09-27 15:07:45 on dist-test-slave-hhpq"
server_key: "89c5e459ed11afabb49e73424e88d84c"
server_key_iv: "a8f4288850baf8c11c8b7f641b5ecf11"
server_key_version: "encryptionkey@0"
I20260927 15:07:45.891702 13682 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260927 15:07:45.894642 13697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:45.896065 13682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 15:07:45.896272 13682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "be770c2746a2448e9afa82fdf76f08ec"
format_stamp: "Formatted at 2026-09-27 15:07:45 on dist-test-slave-hhpq"
server_key: "89c5e459ed11afabb49e73424e88d84c"
server_key_iv: "a8f4288850baf8c11c8b7f641b5ecf11"
server_key_version: "encryptionkey@0"
I20260927 15:07:45.896395 13682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:45.924919 13682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:45.928061 13682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:45.928261 13682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:45.928908 13682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:45.929804 13682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:45.929852 13682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:45.929906 13682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:45.929934 13682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:45.938301 13682 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:40949
I20260927 15:07:45.938333 13810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:40949 every 8 connection(s)
I20260927 15:07:45.939206 13682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:45.939940 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 13682
I20260927 15:07:45.940042 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:07:45.940356 7239 external_mini_cluster.cc:1428] Setting key a3efce73c73b85819eb4596864a2f266
Sep 27 15:07:45 dist-test-slave-hhpq krb5kdc[13307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521665, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM
I20260927 15:07:45.953310 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:45.941155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:39907 (local address 127.7.17.254:33177)
0927 15:07:45.941278 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:45.941282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:45.942401 (+ 1119us) server_negotiation.cc:407] Connection header received
0927 15:07:45.943727 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:45.943730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:45.943776 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:45.943841 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:45.945191 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.945747 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.947009 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.947265 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.947366 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:45.950032 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:45.950036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:45.950038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:45.950065 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:45.951266 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:45.951762 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:45.951767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:45.951769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:45.951846 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:45.952162 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:45.952165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:45.952167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:45.952354 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:45.952439 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:45.953017 (+ 578us) server_negotiation.cc:299] Negotiation successful
0927 15:07:45.953148 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 15:07:45.954314 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:45.941748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.195:39907)
0927 15:07:45.942249 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:45.942311 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:45.943523 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:45.944018 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:45.944025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:45.944385 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:45.945004 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:45.945021 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.945910 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.945913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:45.946886 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:45.946896 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:45.947440 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:45.947444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:45.947663 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:45.948224 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:45.948241 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:45.949843 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:45.951380 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:45.951386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:45.951404 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:45.951665 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:45.951926 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:45.951929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:45.951931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:45.952070 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:45.952450 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:45.952458 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:45.952739 (+ 281us) client_negotiation.cc:769] Sending connection context
0927 15:07:45.952965 (+ 226us) client_negotiation.cc:240] Negotiation successful
0927 15:07:45.953330 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":112,"threads_started":1}
I20260927 15:07:45.955452 13811 heartbeater.cc:347] Connected to a master server at 127.7.17.254:33177
I20260927 15:07:45.955706 13811 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:45.956262 13811 heartbeater.cc:511] Master 127.7.17.254:33177 requested a full tablet report, sending...
I20260927 15:07:45.957322 13351 ts_manager.cc:194] Registered new tserver with Master: be770c2746a2448e9afa82fdf76f08ec (127.7.17.195:40949)
I20260927 15:07:45.957887 13351 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:39907
I20260927 15:07:45.966923 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.7.17.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.17.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.17.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.253.keytab.
Entry for principal oryx/127.7.17.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.253.keytab.
I20260927 15:07:45.994065 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.253:34217
--webserver_interface=127.7.17.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.253.keytab
--principal=oryx/127.7.17.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:33177,127.7.17.253:34217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:46.105649 13816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:46.105947 13816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:46.105986 13816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:46.110018 13816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:46.110085 13816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:46.110102 13816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:46.110117 13816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:46.110132 13816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:46.115402 13816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:33177,127.7.17.253:34217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.253.keytab
--principal=oryx/127.7.17.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.253:34217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.17.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
W20260927 15:07:46.115617 13816 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.7.17.254:33177,127.7.17.253:34217'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 15:07:46.116503 13816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:46.117594 13816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:46.125900 13821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:46.126019 13824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:46.125938 13822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:46.126122 13816 server_base.cc:1034] running on GCE node
I20260927 15:07:46.126683 13816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:46.127760 13816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:46.128899 13816 hybrid_clock.cc:648] HybridClock initialized: now 1790521666128883 us; error 28 us; skew 500 ppm
Sep 27 15:07:46 dist-test-slave-hhpq krb5kdc[13307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521666, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:46.131647 13816 init.cc:377] Logged in from keytab as oryx/127.7.17.253@KRBTEST.COM (short username oryx)
I20260927 15:07:46.132522 13816 webserver.cc:460] Webserver started at http://127.7.17.253:40183/ using document root <none> and password file <none>
I20260927 15:07:46.133111 13816 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:46.133178 13816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:46.133359 13816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:46.135427 13816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/data/instance:
uuid: "3c1aa5b16f0847f2ac9bb445628f78f7"
format_stamp: "Formatted at 2026-09-27 15:07:46 on dist-test-slave-hhpq"
server_key: "3ea05b479c1c4a57d76e78b32c755926"
server_key_iv: "11b329b4b3161d94afede3edc87bc93a"
server_key_version: "encryptionkey@0"
I20260927 15:07:46.135955 13816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/wal/instance:
uuid: "3c1aa5b16f0847f2ac9bb445628f78f7"
format_stamp: "Formatted at 2026-09-27 15:07:46 on dist-test-slave-hhpq"
server_key: "3ea05b479c1c4a57d76e78b32c755926"
server_key_iv: "11b329b4b3161d94afede3edc87bc93a"
server_key_version: "encryptionkey@0"
I20260927 15:07:46.139900 13816 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 15:07:46.142513 13831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:46.143860 13816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:07:46.143986 13816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/wal
uuid: "3c1aa5b16f0847f2ac9bb445628f78f7"
format_stamp: "Formatted at 2026-09-27 15:07:46 on dist-test-slave-hhpq"
server_key: "3ea05b479c1c4a57d76e78b32c755926"
server_key_iv: "11b329b4b3161d94afede3edc87bc93a"
server_key_version: "encryptionkey@0"
I20260927 15:07:46.144090 13816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:46.187351 13816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:46.190567 13816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:46.190757 13816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:46.197556 13816 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.253:34217
I20260927 15:07:46.197825 13883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.253:34217 every 8 connection(s)
I20260927 15:07:46.198549 13816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/data/info.pb
I20260927 15:07:46.200809 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 13816
I20260927 15:07:46.201478 13884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:46.205955 13884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:07:46 dist-test-slave-hhpq krb5kdc[13307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.17.253@KRBTEST.COM
Sep 27 15:07:46 dist-test-slave-hhpq krb5kdc[13307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521666, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.253@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM
I20260927 15:07:46.215473 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.202414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47270 (local address 127.7.17.253:34217)
0927 15:07:46.202812 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:46.202827 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:46.202864 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:07:46.203505 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:46.203523 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:46.203867 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:46.204154 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:46.205052 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.206040 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.207135 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.207720 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.207862 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.210386 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:46.210391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:46.210400 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:46.210426 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:46.212313 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.212738 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.212743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.212747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.212890 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.213152 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.213155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.213157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.213661 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:46.213827 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:46.214226 (+ 399us) server_negotiation.cc:299] Negotiation successful
0927 15:07:46.214510 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":113,"threads_started":1}
I20260927 15:07:46.220323 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.207891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44198 (local address 127.7.17.254:33177)
0927 15:07:46.208017 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:46.208021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:46.208550 (+ 529us) server_negotiation.cc:407] Connection header received
0927 15:07:46.208685 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:46.208688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:46.208734 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:46.208799 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:46.209558 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.210142 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.211152 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.211448 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.211543 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.213680 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:46.213683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:46.213685 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:46.213712 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:46.218696 (+ 4984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.219117 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.219121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.219123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.219201 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.219435 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.219438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.219440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.219650 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:46.219741 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:46.220053 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 15:07:46.220170 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:07:46.220348 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.207812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.0.0.1:44198)
0927 15:07:46.208416 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:46.208471 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:46.208587 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:46.208881 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:46.208887 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:46.209315 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:46.209435 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.209441 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.210263 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.210266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.211036 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.211045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.211553 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.211557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.211664 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.212146 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:46.212158 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:46.213534 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:46.218802 (+ 5268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.218809 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.218815 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.219026 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.219282 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.219285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.219287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.219359 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.219755 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:46.219767 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:46.219871 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 15:07:46.219985 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 15:07:46.220171 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":101,"threads_started":1}
I20260927 15:07:46.221477 13884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:07:46 dist-test-slave-hhpq krb5kdc[13307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521666, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.253@KRBTEST.COM for oryx/127.7.17.253@KRBTEST.COM
I20260927 15:07:46.229496 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.221991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:34217 (local address 127.0.0.1:47286)
0927 15:07:46.222082 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:46.222095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:46.222201 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:46.222407 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:46.222409 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:46.222535 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:46.222624 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.222630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.223995 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.223998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.224539 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.224547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.225097 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.225100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.225203 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.225612 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:46.225624 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:46.226890 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:46.228282 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.228284 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.228285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.228449 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.228672 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.228674 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.228675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.228746 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.229094 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:46.229096 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:46.229201 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 15:07:46.229283 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 15:07:46.229377 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 15:07:46.229606 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.222030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47286 (local address 127.7.17.253:34217)
0927 15:07:46.222146 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:46.222150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:46.222163 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:07:46.222277 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:46.222279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:46.222317 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:46.222385 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:46.222706 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.223876 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.224632 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.225007 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.225109 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.227006 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:46.227010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:46.227011 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:46.227032 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:46.228196 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.228527 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.228530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.228531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.228608 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.228812 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.228814 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.228815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.229013 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:46.229146 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:46.229394 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 15:07:46.229497 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:07:46.232415 13884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7: Bootstrap starting.
I20260927 15:07:46.234721 13884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:46.235440 13884 log.cc:824] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:46.237403 13884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7: No bootstrap required, opened a new log
I20260927 15:07:46.240455 13884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } }
I20260927 15:07:46.240736 13884 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:46.240803 13884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c1aa5b16f0847f2ac9bb445628f78f7, State: Initialized, Role: FOLLOWER
I20260927 15:07:46.241325 13884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } }
I20260927 15:07:46.241952 13890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } } }
I20260927 15:07:46.242273 13890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:07:46.242794 13884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 15:07:46.247725 13901 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:07:46.247864 13901 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:07:46.249821 13816 master_runner.cc:428] Detected that this master 3c1aa5b16f0847f2ac9bb445628f78f7 is joining an existing cluster
I20260927 15:07:46.249878 13816 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.17.253:34217
I20260927 15:07:46.258147 13890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:46.258337 13890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } }
I20260927 15:07:46.259508 13890 leader_election.cc:288] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc7b3a82e94e47b28a0f16ceb4f31721 (127.7.17.254:33177)
I20260927 15:07:46.259994 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.251935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44214 (local address 127.7.17.254:33177)
0927 15:07:46.252052 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:46.252057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:46.252294 (+ 237us) server_negotiation.cc:407] Connection header received
0927 15:07:46.252397 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:46.252400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:46.252447 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:46.252513 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:46.253037 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.253756 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.254904 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.255233 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.255331 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.256465 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:46.256468 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:46.256470 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:46.256494 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:46.258128 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.258660 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.258664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.258666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.258761 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.259045 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.259048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.259050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.259290 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:46.259359 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:46.259697 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 15:07:46.259826 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:07:46.260315 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.251971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47298 (local address 127.7.17.253:34217)
0927 15:07:46.252089 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:46.252093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:46.252554 (+ 461us) server_negotiation.cc:407] Connection header received
0927 15:07:46.252695 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:46.252698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:46.252739 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:46.252812 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:46.253340 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.253994 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.254782 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.255093 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.255187 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.256952 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:46.256956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:46.256958 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:46.256983 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:46.258550 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.259101 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.259104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.259106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.259175 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.259404 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.259407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.259408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.259609 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:46.259677 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:46.260048 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 15:07:46.260171 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42,"spinlock_wait_cycles":2432}
I20260927 15:07:46.260500 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.251894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.0.0.1:44214)
0927 15:07:46.252207 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:46.252221 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:46.252319 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:46.252544 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:46.252547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:46.252770 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:46.252922 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.252929 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.253864 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.253868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.254803 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.254812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.255329 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.255332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.255465 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.255979 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:46.255993 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:46.256328 (+ 335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:46.258252 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.258255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.258258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.258546 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.258864 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.258867 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.258869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.258954 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.259428 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:46.259431 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:46.259524 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 15:07:46.260135 (+ 611us) client_negotiation.cc:240] Negotiation successful
0927 15:07:46.260283 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":21,"spinlock_wait_cycles":6400,"thread_start_us":99,"threads_started":1}
I20260927 15:07:46.260394 13362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" is_pre_election: true
I20260927 15:07:46.261191 13833 leader_election.cc:302] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 3c1aa5b16f0847f2ac9bb445628f78f7; no voters: bc7b3a82e94e47b28a0f16ceb4f31721
I20260927 15:07:46.261562 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.252157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:34217 (local address 127.0.0.1:47298)
0927 15:07:46.252476 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:46.252489 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:46.252609 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:46.252861 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:46.252864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:46.253093 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:46.253231 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.253238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.254109 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.254112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.254681 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.254688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.255209 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.255213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.255321 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.256334 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:46.256348 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:46.256737 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:46.258779 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.258782 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.258784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.258996 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.259254 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.259257 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.259259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.259327 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.259800 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:46.259803 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:46.259885 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 15:07:46.261172 (+ 1287us) client_negotiation.cc:240] Negotiation successful
0927 15:07:46.261397 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":27,"spinlock_wait_cycles":1280,"thread_start_us":98,"threads_started":1}
I20260927 15:07:46.261730 13890 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 27 15:07:46 dist-test-slave-hhpq krb5kdc[13307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521665, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.254@KRBTEST.COM for oryx/127.7.17.253@KRBTEST.COM
I20260927 15:07:46.275259 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.265294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47308 (local address 127.7.17.253:34217)
0927 15:07:46.265399 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:46.265403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:46.265737 (+ 334us) server_negotiation.cc:407] Connection header received
0927 15:07:46.265960 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:46.265963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:46.266018 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:46.266088 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:46.266799 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.267313 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.268278 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.268526 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.268631 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.271517 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:46.271521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:46.271523 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:46.271551 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:46.273170 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.273741 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.273745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.273746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.273838 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.274147 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.274151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.274152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.274460 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:46.274568 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:46.274947 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 15:07:46.275088 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:07:46.275290 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.265255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:34217 (local address 127.0.0.1:47308)
0927 15:07:46.265608 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:46.265659 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:46.265795 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:46.266147 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:46.266153 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:46.266562 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:46.266682 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.266689 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.267417 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.267421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.268138 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.268149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.268697 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.268704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.268865 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.269495 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:46.269510 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:46.271350 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:46.273292 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.273303 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.273311 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.273629 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.273947 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.273950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.273952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.274048 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.274572 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:46.274581 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:46.274723 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 15:07:46.274880 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 15:07:46.275109 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for oryx/127.7.17.252@KRBTEST.COM; defaulting to no policy
I20260927 15:07:46.277318 13351 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.17.253:34217: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7"
member_type: NON_VOTER
last_known_addr {
host: "127.7.17.253"
port: 34217
}
attrs {
promote: true
}
}
dest_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721"
cas_config_opid_index: -1
I20260927 15:07:46.278103 13351 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: NON_VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.17.252@KRBTEST.COM" created.
I20260927 15:07:46.280385 13913 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3c1aa5b16f0847f2ac9bb445628f78f7 (127.7.17.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: NON_VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: true } } }
I20260927 15:07:46.281539 13858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:46.282397 13913 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.17.253:34217
I20260927 15:07:46.282662 13913 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: NON_VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: true } } }
I20260927 15:07:46.282763 13913 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:46.282941 13816 master.cc:537] Master@127.7.17.253:34217 shutting down...
I20260927 15:07:46.282961 13858 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 1 FOLLOWER]: Refusing update from remote peer bc7b3a82e94e47b28a0f16ceb4f31721: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 15:07:46.283726 13912 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: NON_VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:07:46.286286 13339 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 15:07:46.287573 13339 consensus_peers.cc:487] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 -> Peer 3c1aa5b16f0847f2ac9bb445628f78f7 (127.7.17.253:34217): Couldn't send request to peer 3c1aa5b16f0847f2ac9bb445628f78f7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 15:07:46.288316 13816 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:46.288640 13816 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:46.288758 13816 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7: stopping tablet replica
I20260927 15:07:46.294013 13816 master.cc:552] Master@127.7.17.253:34217 shutdown complete.
I20260927 15:07:46.294060 13816 master_runner.cc:305] Clearing existing system tablet
I20260927 15:07:46.294341 13816 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:46.294387 13816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:46.295677 13816 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:46.296671 13919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:46.296949 13816 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:46.297065 13816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:46.297091 13816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/wal
uuid: "3c1aa5b16f0847f2ac9bb445628f78f7"
format_stamp: "Formatted at 2026-09-27 15:07:46 on dist-test-slave-hhpq"
server_key: "3ea05b479c1c4a57d76e78b32c755926"
server_key_iv: "11b329b4b3161d94afede3edc87bc93a"
server_key_version: "encryptionkey@0"
I20260927 15:07:46.298830 13816 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:46.301648 13816 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 15:07:46.301736 13816 log.cc:1196] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 15:07:46.301990 13816 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7: Deleting consensus metadata
I20260927 15:07:46.302134 13816 master_runner.cc:315] Copying system tablet from 127.7.17.254:33177
I20260927 15:07:46.303270 13816 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7: tablet copy: Beginning tablet copy session from remote peer at address 127.7.17.254:33177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.17.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.252.keytab.
Entry for principal oryx/127.7.17.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.252.keytab.
I20260927 15:07:46.305303 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.252:40175
--webserver_interface=127.7.17.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.252.keytab
--principal=oryx/127.7.17.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:33177,127.7.17.253:34217,127.7.17.252:40175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:07:46.312351 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.303945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.7.17.254:33177)
0927 15:07:46.304079 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:46.304083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:46.304173 (+ 90us) server_negotiation.cc:407] Connection header received
0927 15:07:46.304327 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:46.304330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:46.304378 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:46.304447 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:46.305035 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.305866 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.307081 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.307432 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.307537 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.308789 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:46.308794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:46.308797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:46.308877 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:46.310597 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.311100 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.311104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.311106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.311194 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.311469 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.311472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.311473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.311687 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:46.311777 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:46.312067 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 15:07:46.312185 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:07:46.312379 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.303815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.0.0.1:44218)
0927 15:07:46.304079 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:46.304095 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:46.304231 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:46.304496 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:46.304499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:46.304725 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:46.304876 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.304885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.305998 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.306002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.306934 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.306947 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.307554 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.307558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.307651 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.308215 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:46.308230 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:46.308639 (+ 409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:46.310716 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.310720 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.310723 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.310992 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.311281 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.311284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.311286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.311383 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.311778 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:46.311781 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:46.311895 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 15:07:46.312008 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 15:07:46.312182 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":91,"threads_started":1}
I20260927 15:07:46.313338 13372 tablet_copy_service.cc:140] P bc7b3a82e94e47b28a0f16ceb4f31721: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3c1aa5b16f0847f2ac9bb445628f78f7 ({username='oryx', principal='oryx/127.7.17.253@KRBTEST.COM'} at 127.0.0.1:44218)
I20260927 15:07:46.313508 13372 tablet_copy_service.cc:161] P bc7b3a82e94e47b28a0f16ceb4f31721: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3c1aa5b16f0847f2ac9bb445628f78f7 at {username='oryx', principal='oryx/127.7.17.253@KRBTEST.COM'} at 127.0.0.1:44218: session id = 3c1aa5b16f0847f2ac9bb445628f78f7-00000000000000000000000000000000
I20260927 15:07:46.315469 13372 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 15:07:46.317427 13816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 15:07:46.319861 13816 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7: tablet copy: Starting download of 0 data blocks...
I20260927 15:07:46.320077 13816 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7: tablet copy: Starting download of 1 WAL segments...
I20260927 15:07:46.322710 13816 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 15:07:46.324751 13372 tablet_copy_service.cc:342] P bc7b3a82e94e47b28a0f16ceb4f31721: Request end of tablet copy session 3c1aa5b16f0847f2ac9bb445628f78f7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.17.253@KRBTEST.COM'} at 127.0.0.1:44218
I20260927 15:07:46.324949 13372 tablet_copy_service.cc:433] P bc7b3a82e94e47b28a0f16ceb4f31721: ending tablet copy session 3c1aa5b16f0847f2ac9bb445628f78f7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3c1aa5b16f0847f2ac9bb445628f78f7
I20260927 15:07:46.332458 13816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:46.334168 13927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:46.334369 13930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:46.334939 13816 server_base.cc:1034] running on GCE node
W20260927 15:07:46.334980 13928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:46.335281 13816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:46.335590 13816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:46.336741 13816 hybrid_clock.cc:648] HybridClock initialized: now 1790521666336731 us; error 28 us; skew 500 ppm
Sep 27 15:07:46 dist-test-slave-hhpq krb5kdc[13307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521666, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:46.338320 13816 init.cc:377] Logged in from keytab as oryx/127.7.17.253@KRBTEST.COM (short username oryx)
I20260927 15:07:46.338783 13816 webserver.cc:460] Webserver started at http://127.7.17.253:45857/ using document root <none> and password file <none>
I20260927 15:07:46.338975 13816 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:46.339015 13816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:46.349311 13816 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.001s sys 0.000s
I20260927 15:07:46.350519 13937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:46.350739 13816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 15:07:46.350797 13816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/wal
uuid: "3c1aa5b16f0847f2ac9bb445628f78f7"
format_stamp: "Formatted at 2026-09-27 15:07:46 on dist-test-slave-hhpq"
server_key: "3ea05b479c1c4a57d76e78b32c755926"
server_key_iv: "11b329b4b3161d94afede3edc87bc93a"
server_key_version: "encryptionkey@0"
I20260927 15:07:46.350883 13816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:46.387920 13816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:46.390007 13816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:46.394555 13816 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.253:34217
I20260927 15:07:46.394680 13989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.253:34217 every 8 connection(s)
I20260927 15:07:46.395120 13816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-1/data/info.pb
I20260927 15:07:46.397013 13990 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:07:46.398085 13990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7: Bootstrap starting.
I20260927 15:07:46.413818 13990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:07:46.414173 13990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7: Bootstrap complete.
I20260927 15:07:46.414798 13990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: NON_VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: true } }
I20260927 15:07:46.414932 13990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3c1aa5b16f0847f2ac9bb445628f78f7, State: Initialized, Role: LEARNER
I20260927 15:07:46.415038 13990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: NON_VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: true } }
I20260927 15:07:46.415531 13990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:46.421126 13993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: NON_VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: true } } }
I20260927 15:07:46.421262 13993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [sys.catalog]: This master's current role is: LEARNER
I20260927 15:07:46.418781 14004 catalog_manager.cc:1259] Loaded cluster ID: 79c4dfdb3ecf46408724e71fcbeb7be8
I20260927 15:07:46.421974 14004 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7: loading cluster ID for follower catalog manager: success
I20260927 15:07:46.424283 14004 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7: acquiring CA information for follower catalog manager: success
I20260927 15:07:46.425757 14004 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 15:07:46.436401 13926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:46.436704 13926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:46.436745 13926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:46.440891 13926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:46.440963 13926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:46.440982 13926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:46.440999 13926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:46.441012 13926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:46.446636 13926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:33177,127.7.17.253:34217,127.7.17.252:40175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.252.keytab
--principal=oryx/127.7.17.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.252:40175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.17.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:46.447690 13926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:46.448690 13926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:46.457530 14010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:46.457541 14009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:46.457599 14012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:46.457769 13926 server_base.cc:1034] running on GCE node
I20260927 15:07:46.458393 13926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:46.459643 13926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:46.460860 13926 hybrid_clock.cc:648] HybridClock initialized: now 1790521666460808 us; error 45 us; skew 500 ppm
Sep 27 15:07:46 dist-test-slave-hhpq krb5kdc[13307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521666, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:46.463817 13926 init.cc:377] Logged in from keytab as oryx/127.7.17.252@KRBTEST.COM (short username oryx)
I20260927 15:07:46.464877 13926 webserver.cc:460] Webserver started at http://127.7.17.252:32907/ using document root <none> and password file <none>
I20260927 15:07:46.465433 13926 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:46.465492 13926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:46.465710 13926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:46.467811 13926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/data/instance:
uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c"
format_stamp: "Formatted at 2026-09-27 15:07:46 on dist-test-slave-hhpq"
server_key: "ad9c6fa0538da879c4c14a83743c7a89"
server_key_iv: "df3338db1d178f84bcfa3d948a888bbe"
server_key_version: "encryptionkey@0"
I20260927 15:07:46.468400 13926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/wal/instance:
uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c"
format_stamp: "Formatted at 2026-09-27 15:07:46 on dist-test-slave-hhpq"
server_key: "ad9c6fa0538da879c4c14a83743c7a89"
server_key_iv: "df3338db1d178f84bcfa3d948a888bbe"
server_key_version: "encryptionkey@0"
I20260927 15:07:46.472711 13926 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 15:07:46.475497 14019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:46.477025 13926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:46.477152 13926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/wal
uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c"
format_stamp: "Formatted at 2026-09-27 15:07:46 on dist-test-slave-hhpq"
server_key: "ad9c6fa0538da879c4c14a83743c7a89"
server_key_iv: "df3338db1d178f84bcfa3d948a888bbe"
server_key_version: "encryptionkey@0"
I20260927 15:07:46.477267 13926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:46.499419 13926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:46.502708 13926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:46.502883 13926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:46.510353 13926 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.252:40175
I20260927 15:07:46.510643 14071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.252:40175 every 8 connection(s)
I20260927 15:07:46.511329 13926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/data/info.pb
I20260927 15:07:46.512877 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 13926
I20260927 15:07:46.515131 14072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:46.519049 14072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:07:46 dist-test-slave-hhpq krb5kdc[13307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.17.252@KRBTEST.COM
I20260927 15:07:46.525630 13543 heartbeater.cc:503] Master 127.7.17.254:33177 was elected leader, sending a full tablet report...
Sep 27 15:07:46 dist-test-slave-hhpq krb5kdc[13307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521666, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.252@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM
I20260927 15:07:46.527444 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.514274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60028 (local address 127.7.17.252:40175)
0927 15:07:46.514632 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:46.514647 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:46.514681 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:07:46.515347 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:46.515365 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:46.515840 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:46.516169 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:46.517041 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.518079 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.518919 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.519397 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.519547 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.522425 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:46.522432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:46.522444 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:46.522476 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:46.524229 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.524630 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.524636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.524642 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.524781 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.525074 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.525077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.525079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.525596 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:46.525848 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:46.526080 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 15:07:46.526387 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"spinlock_wait_cycles":7552,"thread_start_us":98,"threads_started":1}
I20260927 15:07:46.529949 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.520892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44234 (local address 127.7.17.254:33177)
0927 15:07:46.521009 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:46.521013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:46.521366 (+ 353us) server_negotiation.cc:407] Connection header received
0927 15:07:46.521487 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:46.521489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:46.521545 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:46.521620 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:46.522448 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.523180 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.524187 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.524467 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.524544 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.526917 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:46.526921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:46.526924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:46.526948 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:46.528241 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.528706 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.528710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.528712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.528794 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.529061 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.529064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.529066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.529257 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:46.529345 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:46.529668 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 15:07:46.529792 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 15:07:46.529987 14075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.520764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.0.0.1:44234)
0927 15:07:46.521197 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:46.521262 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:46.521402 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:46.521689 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:46.521695 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:46.522154 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:46.522323 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.522330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.523298 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.523301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.524059 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.524069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.524571 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.524575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.524689 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.525288 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:46.525300 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:46.526766 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:46.528337 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.528344 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.528349 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.528617 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.528887 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.528890 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.528892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.528970 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.529357 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:46.529363 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:46.529472 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 15:07:46.529601 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 15:07:46.529793 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"spinlock_wait_cycles":1280,"thread_start_us":181,"threads_started":1}
I20260927 15:07:46.531250 14072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:07:46 dist-test-slave-hhpq krb5kdc[13307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521666, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.252@KRBTEST.COM for oryx/127.7.17.253@KRBTEST.COM
I20260927 15:07:46.539356 14075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.531710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:34217 (local address 127.0.0.1:47310)
0927 15:07:46.531844 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:46.531857 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:46.531922 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:46.532353 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:46.532356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:46.532513 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:46.532607 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.532613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.533569 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.533572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.534102 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.534109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.534534 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.534538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.534631 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.535186 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:46.535197 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:46.536343 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:46.537961 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.537964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.537966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.538145 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.538390 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.538393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.538395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.538469 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.538823 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:46.538826 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:46.538913 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 15:07:46.539022 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 15:07:46.539163 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 15:07:46.539356 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.531892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47310 (local address 127.7.17.253:34217)
0927 15:07:46.532158 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:46.532162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:46.532173 (+ 11us) server_negotiation.cc:407] Connection header received
0927 15:07:46.532206 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:46.532208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:46.532248 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:46.532323 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:46.532718 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.533464 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.534204 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.534440 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.534527 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.536473 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:46.536476 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:46.536478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:46.536502 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:46.537862 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.538230 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.538234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.538235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.538311 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.538545 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.538548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.538550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.538739 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:46.538822 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:46.539075 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 15:07:46.539189 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":108,"threads_started":1}
I20260927 15:07:46.540246 14072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:07:46 dist-test-slave-hhpq krb5kdc[13307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521666, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.252@KRBTEST.COM for oryx/127.7.17.252@KRBTEST.COM
I20260927 15:07:46.548434 14075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.540850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:40175 (local address 127.0.0.1:60034)
0927 15:07:46.540949 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:46.540963 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:46.541034 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:46.541251 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:46.541253 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:46.541407 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:46.541501 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.541506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.542639 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.542642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.543171 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.543178 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.543589 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.543592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.543648 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.544071 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:46.544137 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:46.545421 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:46.546979 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.546981 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.546983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.547182 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.547477 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.547480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.547481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.547571 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.547925 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:46.547927 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:46.548010 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 15:07:46.548148 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 15:07:46.548261 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 15:07:46.548446 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.540909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60034 (local address 127.7.17.252:40175)
0927 15:07:46.541031 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:46.541036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:46.541050 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:07:46.541085 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:46.541095 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:46.541135 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:46.541204 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:46.541627 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.542514 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.543283 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.543505 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.543581 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.545571 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:46.545575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:46.545578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:46.545599 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:46.546858 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.547294 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.547297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.547298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.547375 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.547659 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.547663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.547664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.547843 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:46.547918 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:46.548228 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 15:07:46.548325 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 15:07:46.551591 14072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c: Bootstrap starting.
I20260927 15:07:46.554026 14072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:46.554790 14072 log.cc:824] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:46.556913 14072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c: No bootstrap required, opened a new log
I20260927 15:07:46.560027 14072 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } }
I20260927 15:07:46.560278 14072 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:46.560340 14072 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7a6e46bfaba4e33802c1991a2cc8c0c, State: Initialized, Role: FOLLOWER
I20260927 15:07:46.560786 14072 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } }
I20260927 15:07:46.561347 14078 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } } }
I20260927 15:07:46.561493 14078 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:07:46.562151 14072 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [sys.catalog]: configured and running, proceeding with master startup.
W20260927 15:07:46.566910 14089 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:07:46.566995 14089 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:07:46.570586 13926 master_runner.cc:428] Detected that this master d7a6e46bfaba4e33802c1991a2cc8c0c is joining an existing cluster
I20260927 15:07:46.570649 13926 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.17.252:40175
I20260927 15:07:46.581108 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 13414
I20260927 15:07:46.588539 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:45549
--local_ip_for_outbound_sockets=127.7.17.193
--tserver_master_addrs=127.7.17.254:33177
--webserver_port=44421
--webserver_interface=127.7.17.193
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.193.keytab
--principal=oryx/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.17.254:33177,127.7.17.253:34217
--tserver_master_addrs=127.7.17.254:33177,127.7.17.253:34217,127.7.17.252:40175 with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:07:46.606105 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.573330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:40175 (local address 127.0.0.1:60048)
0927 15:07:46.573694 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:46.573707 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:46.573827 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:46.574129 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:46.574132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:46.574455 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:46.574605 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.574613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.575593 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.575595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.576712 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.576722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.577377 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.577380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.577460 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.577994 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:46.578024 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:46.601585 (+ 23561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:46.603946 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.603949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.603951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.604163 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.604400 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.604402 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.604404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.604490 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.605134 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:46.605137 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:46.605244 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 15:07:46.605323 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 15:07:46.605471 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":168,"spinlock_wait_cycles":8320,"thread_start_us":172,"threads_started":1}
I20260927 15:07:46.607048 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.573393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60048 (local address 127.7.17.252:40175)
0927 15:07:46.573478 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:46.573482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:46.573771 (+ 289us) server_negotiation.cc:407] Connection header received
0927 15:07:46.573913 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:46.573916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:46.573952 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:46.573995 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:46.574718 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.575473 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.576854 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.577147 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.577218 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.601794 (+ 24576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:46.601798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:46.601800 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:46.601823 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:46.603817 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.604246 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.604249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.604251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.604314 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.604572 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.604575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.604577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.604807 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:46.606733 (+ 1926us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:46.606802 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 15:07:46.606914 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 15:07:46.608815 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.572755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47312 (local address 127.7.17.253:34217)
0927 15:07:46.572890 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:46.572894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:46.573743 (+ 849us) server_negotiation.cc:407] Connection header received
0927 15:07:46.573905 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:46.573908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:46.573964 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:46.574013 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:46.574734 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.575451 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.576895 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.577251 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.577314 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.602532 (+ 25218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:46.602537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:46.602540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:46.602596 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:46.604060 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.605018 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.605022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.605024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.605133 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.606860 (+ 1727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.606864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.606865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.607110 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:46.607206 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:46.608547 (+ 1341us) server_negotiation.cc:299] Negotiation successful
0927 15:07:46.608698 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 15:07:46.609388 14097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.572970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:34217 (local address 127.0.0.1:47312)
0927 15:07:46.573663 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:46.573676 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:46.573804 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:46.574140 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:46.574142 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:46.574464 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:46.574608 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.574615 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.575582 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.575586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.576736 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.576742 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.577465 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.577468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.577633 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.578340 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:46.578354 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:46.602331 (+ 23977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:46.604192 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.604195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.604197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.604406 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.605572 (+ 1166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.605574 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.605576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.605693 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.607495 (+ 1802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:46.607498 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:46.607591 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 15:07:46.609021 (+ 1430us) client_negotiation.cc:240] Negotiation successful
0927 15:07:46.609223 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":223,"thread_start_us":108,"threads_started":1}
I20260927 15:07:46.609472 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.572788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44250 (local address 127.7.17.254:33177)
0927 15:07:46.572919 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:46.572923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:46.573323 (+ 400us) server_negotiation.cc:407] Connection header received
0927 15:07:46.573527 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:46.573530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:46.573583 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:46.573638 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:46.574321 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.575118 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.576897 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.577236 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.577313 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.604622 (+ 27309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:46.604627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:46.604630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:46.604661 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:46.606378 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.607287 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.607291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.607292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.607390 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.607897 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.607900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.607902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.608142 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:46.608222 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:46.609224 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0927 15:07:46.609339 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:07:46.609952 14096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.572606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.0.0.1:44250)
0927 15:07:46.573242 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:46.573256 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:46.573410 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:46.573821 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:46.573824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:46.574060 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:46.574189 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.574197 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.575247 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.575250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.576730 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.576747 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.577465 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.577468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.577686 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.601556 (+ 23870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:46.601576 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:46.603398 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:46.606524 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.606526 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.606528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.606749 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.607712 (+ 963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.607714 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.607716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.607793 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.608273 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:46.608276 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:46.608363 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 15:07:46.609657 (+ 1294us) client_negotiation.cc:240] Negotiation successful
0927 15:07:46.609795 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":244,"spinlock_wait_cycles":649344,"thread_start_us":98,"threads_started":1}
Sep 27 15:07:46 dist-test-slave-hhpq krb5kdc[13307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521665, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.254@KRBTEST.COM for oryx/127.7.17.252@KRBTEST.COM
I20260927 15:07:46.621181 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.613616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:40175 (local address 127.0.0.1:60056)
0927 15:07:46.613793 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:46.613806 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:46.613892 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:46.614149 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:46.614152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:46.614327 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:46.614414 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.614420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.615125 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.615128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.615671 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.615680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.616277 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.616281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.616380 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.616902 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:46.616917 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:46.618190 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:46.619739 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.619742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.619744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.619941 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.620186 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.620189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.620191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.620271 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.620628 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:46.620631 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:46.620728 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 15:07:46.620815 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 15:07:46.621008 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 15:07:46.621181 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.613670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60056 (local address 127.7.17.252:40175)
0927 15:07:46.613777 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:46.613781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:46.613848 (+ 67us) server_negotiation.cc:407] Connection header received
0927 15:07:46.614005 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:46.614007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:46.614053 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:46.614118 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:46.614514 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.615022 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.615801 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.616163 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.616257 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.618323 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:46.618326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:46.618328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:46.618354 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:46.619621 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.620027 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.620031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.620032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.620105 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.620348 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.620351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.620353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.620534 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:46.620627 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:46.620905 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 15:07:46.621024 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:07:46.622437 13351 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.17.252:40175: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c"
member_type: NON_VOTER
last_known_addr {
host: "127.7.17.252"
port: 40175
}
attrs {
promote: true
}
}
dest_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721"
cas_config_opid_index: 5
I20260927 15:07:46.622991 13351 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: NON_VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: true } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: NON_VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: true } }
I20260927 15:07:46.624029 13912 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d7a6e46bfaba4e33802c1991a2cc8c0c (127.7.17.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: NON_VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: true } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: NON_VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: true } } }
I20260927 15:07:46.625114 14046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:46.625643 13912 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.17.252:40175
I20260927 15:07:46.625823 13912 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: NON_VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: true } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: NON_VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: true } } }
I20260927 15:07:46.625914 13912 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:46.626540 14046 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 1 FOLLOWER]: Refusing update from remote peer bc7b3a82e94e47b28a0f16ceb4f31721: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 15:07:46.627149 13926 master.cc:537] Master@127.7.17.252:40175 shutting down...
I20260927 15:07:46.627149 13915 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: NON_VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:07:46.629536 13339 consensus_peers.cc:487] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 -> Peer d7a6e46bfaba4e33802c1991a2cc8c0c (127.7.17.252:40175): Couldn't send request to peer d7a6e46bfaba4e33802c1991a2cc8c0c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 15:07:46.633769 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.624138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:34217 (local address 127.0.0.1:47314)
0927 15:07:46.629112 (+ 4974us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:46.629128 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:46.629232 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:46.629753 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:46.629756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:46.629776 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:46.629875 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.629881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.630907 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.630911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.632509 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.632521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.633288 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.633292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.633382 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.633435 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 15:07:46.633523 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 15:07:46.633610 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4898}
I20260927 15:07:46.634223 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.624444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47314 (local address 127.7.17.253:34217)
0927 15:07:46.628999 (+ 4555us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:46.629004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:46.629316 (+ 312us) server_negotiation.cc:407] Connection header received
0927 15:07:46.629370 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:46.629373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:46.629435 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:46.629514 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:46.630000 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.630779 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.632670 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.633170 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.633268 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.633443 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:46.634010 (+ 567us) server_negotiation.cc:299] Negotiation successful
0927 15:07:46.634103 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4451}
I20260927 15:07:46.634500 13926 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:46.634806 13926 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:46.634869 13957 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 1 LEARNER]: Refusing update from remote peer bc7b3a82e94e47b28a0f16ceb4f31721: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 15:07:46.634919 13926 tablet_replica.cc:330] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c: stopping tablet replica
I20260927 15:07:46.635324 13915 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: NON_VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 15:07:46.637341 13957 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d7a6e46bfaba4e33802c1991a2cc8c0c (127.7.17.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: NON_VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: true } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: NON_VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: true } } }
I20260927 15:07:46.639714 13993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc7b3a82e94e47b28a0f16ceb4f31721. Latest consensus state: current_term: 1 leader_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: NON_VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: true } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: NON_VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: true } } }
I20260927 15:07:46.639824 13993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [sys.catalog]: This master's current role is: LEARNER
I20260927 15:07:46.639915 13993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: NON_VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: true } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: NON_VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: true } } }
I20260927 15:07:46.639981 13993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [sys.catalog]: This master's current role is: LEARNER
I20260927 15:07:46.642143 13912 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721: attempting to promote NON_VOTER 3c1aa5b16f0847f2ac9bb445628f78f7 to VOTER
I20260927 15:07:46.642591 13912 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: NON_VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: true } }
I20260927 15:07:46.643930 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.643550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:40175 (local address 127.0.0.1:60064)
0927 15:07:46.643665 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:46.643839 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:40175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 15:07:46.644532 13957 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 1 LEARNER]: Refusing update from remote peer bc7b3a82e94e47b28a0f16ceb4f31721: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 15:07:46.644579 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.644348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:40175 (local address 127.0.0.1:60070)
0927 15:07:46.644452 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:46.644501 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:40175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 15:07:46.644657 13339 consensus_peers.cc:487] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 -> Peer d7a6e46bfaba4e33802c1991a2cc8c0c (127.7.17.252:40175): Couldn't send request to peer d7a6e46bfaba4e33802c1991a2cc8c0c. Status: Network error: Client connection negotiation failed: client connection to 127.7.17.252:40175: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 15:07:46.645073 13915 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 15:07:46.646835 13913 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3c1aa5b16f0847f2ac9bb445628f78f7 (127.7.17.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: NON_VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: true } } }
I20260927 15:07:46.647336 13957 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3c1aa5b16f0847f2ac9bb445628f78f7 (127.7.17.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: NON_VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: true } } }
I20260927 15:07:46.648414 13913 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: NON_VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: true } } }
I20260927 15:07:46.648518 13913 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:46.648641 13913 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: NON_VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: true } } }
I20260927 15:07:46.648713 13913 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:46.648991 14100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: NON_VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: true } } }
I20260927 15:07:46.649135 14100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:07:46.651700 13926 master.cc:552] Master@127.7.17.252:40175 shutdown complete.
I20260927 15:07:46.651744 13926 master_runner.cc:305] Clearing existing system tablet
I20260927 15:07:46.651994 13926 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:46.652045 13926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:46.653152 13926 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 15:07:46.653977 14102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:46.654192 13926 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:46.654321 13926 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:46.654358 13926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/wal
uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c"
format_stamp: "Formatted at 2026-09-27 15:07:46 on dist-test-slave-hhpq"
server_key: "ad9c6fa0538da879c4c14a83743c7a89"
server_key_iv: "df3338db1d178f84bcfa3d948a888bbe"
server_key_version: "encryptionkey@0"
I20260927 15:07:46.656096 13926 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:46.658077 13926 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 15:07:46.658159 13926 log.cc:1196] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 15:07:46.658363 13926 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c: Deleting consensus metadata
I20260927 15:07:46.658568 13926 master_runner.cc:315] Copying system tablet from 127.7.17.254:33177
I20260927 15:07:46.659578 13926 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c: tablet copy: Beginning tablet copy session from remote peer at address 127.7.17.254:33177
I20260927 15:07:46.667488 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.660167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44254 (local address 127.7.17.254:33177)
0927 15:07:46.660298 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:46.660303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:46.660467 (+ 164us) server_negotiation.cc:407] Connection header received
0927 15:07:46.660583 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:46.660587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:46.660637 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:46.660706 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:46.661410 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.661931 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.662985 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.663333 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.663426 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.664587 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:46.664591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:46.664594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:46.664623 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:46.665986 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.666368 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.666371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.666373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.666453 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.666702 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.666706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.666708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.666902 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:46.666985 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:46.667230 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 15:07:46.667336 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:07:46.667505 14108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.660086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.0.0.1:44254)
0927 15:07:46.660375 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:46.660389 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:46.660498 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:46.660730 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:46.660733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:46.660974 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:46.661282 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.661288 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.662061 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.662065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.662857 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.662868 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.663436 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.663439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.663529 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.664104 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:46.664119 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:46.664450 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:46.666081 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.666084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.666087 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.666280 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.666530 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.666533 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.666535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.666622 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.666986 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:46.666989 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:46.667082 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 15:07:46.667170 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 15:07:46.667314 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":83,"threads_started":1}
I20260927 15:07:46.668272 13372 tablet_copy_service.cc:140] P bc7b3a82e94e47b28a0f16ceb4f31721: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d7a6e46bfaba4e33802c1991a2cc8c0c ({username='oryx', principal='oryx/127.7.17.252@KRBTEST.COM'} at 127.0.0.1:44254)
I20260927 15:07:46.668367 13372 tablet_copy_service.cc:161] P bc7b3a82e94e47b28a0f16ceb4f31721: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d7a6e46bfaba4e33802c1991a2cc8c0c at {username='oryx', principal='oryx/127.7.17.252@KRBTEST.COM'} at 127.0.0.1:44254: session id = d7a6e46bfaba4e33802c1991a2cc8c0c-00000000000000000000000000000000
I20260927 15:07:46.669368 13372 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 15:07:46.670531 13926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 15:07:46.672432 13926 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c: tablet copy: Starting download of 0 data blocks...
I20260927 15:07:46.672608 13926 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c: tablet copy: Starting download of 1 WAL segments...
I20260927 15:07:46.673941 13926 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 15:07:46.675521 13372 tablet_copy_service.cc:342] P bc7b3a82e94e47b28a0f16ceb4f31721: Request end of tablet copy session d7a6e46bfaba4e33802c1991a2cc8c0c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.17.252@KRBTEST.COM'} at 127.0.0.1:44254
I20260927 15:07:46.675622 13372 tablet_copy_service.cc:433] P bc7b3a82e94e47b28a0f16ceb4f31721: ending tablet copy session d7a6e46bfaba4e33802c1991a2cc8c0c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d7a6e46bfaba4e33802c1991a2cc8c0c
I20260927 15:07:46.682209 13926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:46.684029 14109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:46.684218 14112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:46.684540 13926 server_base.cc:1034] running on GCE node
W20260927 15:07:46.684589 14110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:46.684885 13926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:46.685180 13926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:46.686312 13926 hybrid_clock.cc:648] HybridClock initialized: now 1790521666686307 us; error 26 us; skew 500 ppm
Sep 27 15:07:46 dist-test-slave-hhpq krb5kdc[13307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521666, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:46.687850 13926 init.cc:377] Logged in from keytab as oryx/127.7.17.252@KRBTEST.COM (short username oryx)
I20260927 15:07:46.688330 13926 webserver.cc:460] Webserver started at http://127.7.17.252:46733/ using document root <none> and password file <none>
I20260927 15:07:46.688511 13926 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:46.688546 13926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:46.689697 13926 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:46.690536 14119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:46.690759 13926 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:46.690828 13926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/wal
uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c"
format_stamp: "Formatted at 2026-09-27 15:07:46 on dist-test-slave-hhpq"
server_key: "ad9c6fa0538da879c4c14a83743c7a89"
server_key_iv: "df3338db1d178f84bcfa3d948a888bbe"
server_key_version: "encryptionkey@0"
I20260927 15:07:46.690912 13926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:46.724591 13926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:46.726536 13926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:46.731101 13926 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.252:40175
I20260927 15:07:46.731252 14171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.252:40175 every 8 connection(s)
I20260927 15:07:46.731666 13926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/master-2/data/info.pb
I20260927 15:07:46.733160 14172 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:07:46.734007 14172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c: Bootstrap starting.
I20260927 15:07:46.738715 13677 heartbeater.cc:503] Master 127.7.17.254:33177 was elected leader, sending a full tablet report...
I20260927 15:07:46.743608 14172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:07:46.743899 14172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c: Bootstrap complete.
I20260927 15:07:46.744568 14172 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: NON_VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: true } }
I20260927 15:07:46.744706 14172 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d7a6e46bfaba4e33802c1991a2cc8c0c, State: Initialized, Role: LEARNER
I20260927 15:07:46.744817 14172 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: NON_VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: true } }
I20260927 15:07:46.745182 14178 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: NON_VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: true } } }
I20260927 15:07:46.745334 14172 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:46.745378 14178 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [sys.catalog]: This master's current role is: LEARNER
I20260927 15:07:46.749356 14189 catalog_manager.cc:1259] Loaded cluster ID: 79c4dfdb3ecf46408724e71fcbeb7be8
I20260927 15:07:46.749413 14189 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c: loading cluster ID for follower catalog manager: success
I20260927 15:07:46.751302 14189 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c: acquiring CA information for follower catalog manager: success
W20260927 15:07:46.751071 14098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:46.751386 14098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:46.751446 14098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 15:07:46.752601 14189 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 15:07:46.757756 14098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:46.757838 14098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:46.757926 14098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:07:46.765318 14098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.193.keytab
--principal=oryx/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:45549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=44421
--tserver_master_addrs=127.7.17.254:33177,127.7.17.253:34217,127.7.17.252:40175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:46.766448 14098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:46.767436 14098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:46.777271 14192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:46.777472 14098 server_base.cc:1034] running on GCE node
W20260927 15:07:46.777287 14191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:46.777304 14194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:46.778082 14098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:46.778817 14098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:46.779997 14098 hybrid_clock.cc:648] HybridClock initialized: now 1790521666779974 us; error 37 us; skew 500 ppm
Sep 27 15:07:46 dist-test-slave-hhpq krb5kdc[13307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521666, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:46.782953 14098 init.cc:377] Logged in from keytab as oryx/127.7.17.193@KRBTEST.COM (short username oryx)
I20260927 15:07:46.783999 14098 webserver.cc:460] Webserver started at http://127.7.17.193:44421/ using document root <none> and password file <none>
I20260927 15:07:46.784628 14098 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:46.784713 14098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:46.789016 14098 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 15:07:46.791115 14201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:46.792446 14098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:07:46.792559 14098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "627b253d79314dd39ba841bfedb233e6"
format_stamp: "Formatted at 2026-09-27 15:07:45 on dist-test-slave-hhpq"
server_key: "e4c9a1f7c5d5f2f7d4f337d5c47bce2f"
server_key_iv: "22eead3a68f39c61cfec8b4fd58eeb66"
server_key_version: "encryptionkey@0"
I20260927 15:07:46.793149 14098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:46.819248 14098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:46.822456 14098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:46.822635 14098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:46.823285 14098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:46.824153 14098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:46.824209 14098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:46.824261 14098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:46.824287 14098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:46.833364 14098 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:45549
I20260927 15:07:46.833396 14314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:45549 every 8 connection(s)
I20260927 15:07:46.834417 14098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:46.839406 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 14098
I20260927 15:07:46.839607 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 13548
I20260927 15:07:46.846773 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:46871
--local_ip_for_outbound_sockets=127.7.17.194
--tserver_master_addrs=127.7.17.254:33177
--webserver_port=41513
--webserver_interface=127.7.17.194
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.194.keytab
--principal=oryx/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.17.254:33177,127.7.17.253:34217
--tserver_master_addrs=127.7.17.254:33177,127.7.17.253:34217,127.7.17.252:40175 with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:07:46 dist-test-slave-hhpq krb5kdc[13307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521666, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.253@KRBTEST.COM
Sep 27 15:07:46 dist-test-slave-hhpq krb5kdc[13307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521666, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM
Sep 27 15:07:46 dist-test-slave-hhpq krb5kdc[13307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521666, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.252@KRBTEST.COM
I20260927 15:07:46.853238 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.838204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:40909 (local address 127.7.17.253:34217)
0927 15:07:46.838323 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:46.838327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:46.838460 (+ 133us) server_negotiation.cc:407] Connection header received
0927 15:07:46.838896 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:46.838899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:46.838958 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:46.839029 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:46.841447 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.842047 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.843998 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.844355 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.844432 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.848617 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:46.848622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:46.848624 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:46.848656 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:46.850356 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.851285 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.851288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.851290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.851383 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.851779 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.851783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.851784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.852019 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:46.852092 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:46.852936 (+ 844us) server_negotiation.cc:299] Negotiation successful
0927 15:07:46.853065 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:07:46.853297 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.836301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:34185 (local address 127.7.17.252:40175)
0927 15:07:46.836540 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:46.836543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:46.837697 (+ 1154us) server_negotiation.cc:407] Connection header received
0927 15:07:46.838927 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:46.838930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:46.838990 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:46.839050 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:46.842813 (+ 3763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.843945 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.845169 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.845490 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.845566 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.848596 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:46.848600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:46.848603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:46.848633 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:46.850525 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.851278 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.851282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.851283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.851374 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.851766 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.851769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.851771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.852025 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:46.852110 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:46.852919 (+ 809us) server_negotiation.cc:299] Negotiation successful
0927 15:07:46.853055 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":79,"threads_started":1}
I20260927 15:07:46.854681 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.836301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:48403 (local address 127.7.17.254:33177)
0927 15:07:46.836432 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:46.836436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:46.837701 (+ 1265us) server_negotiation.cc:407] Connection header received
0927 15:07:46.839013 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:46.839017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:46.839090 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:46.839160 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:46.841445 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.842270 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.843998 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.844355 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.844443 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.848596 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:46.848600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:46.848603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:46.848633 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:46.851066 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.851680 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.851683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.851685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.851789 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:46.852143 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:46.852146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:46.852148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:46.852351 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:46.852431 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:46.854409 (+ 1978us) server_negotiation.cc:299] Negotiation successful
0927 15:07:46.854529 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:07:46.854730 14322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.838098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:34217 (local address 127.7.17.193:40909)
0927 15:07:46.838373 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:46.838387 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:46.838691 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:46.839232 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:46.839242 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:46.840355 (+ 1113us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:46.841247 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.841272 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.842208 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.842211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.843847 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.843858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.844620 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.844624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.845014 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.845751 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:46.845779 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:46.848368 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:46.850621 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.850629 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.850652 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.851158 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.851498 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.851501 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.851502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.851670 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.852176 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:46.852183 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:46.852598 (+ 415us) client_negotiation.cc:769] Sending connection context
0927 15:07:46.853448 (+ 850us) client_negotiation.cc:240] Negotiation successful
0927 15:07:46.853606 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":269,"spinlock_wait_cycles":36992,"thread_start_us":93,"threads_started":1}
I20260927 15:07:46.855010 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.836932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:40175 (local address 127.7.17.193:34185)
0927 15:07:46.837513 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:46.837579 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:46.838683 (+ 1104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:46.839232 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:46.839242 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:46.842445 (+ 3203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:46.842663 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.842669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.844107 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.844110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.845022 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.845032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.845739 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.845742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.845838 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.846286 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:46.846300 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:46.848437 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:46.850653 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.850656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.850658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.851150 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.851482 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.851485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.851487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.851669 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.852176 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:46.852183 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:46.852595 (+ 412us) client_negotiation.cc:769] Sending connection context
0927 15:07:46.853546 (+ 951us) client_negotiation.cc:240] Negotiation successful
0927 15:07:46.853677 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":88,"spinlock_wait_cycles":29312,"thread_start_us":151,"threads_started":1}
I20260927 15:07:46.855535 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:46.836932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.193:48403)
0927 15:07:46.837513 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:46.837579 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:46.838690 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:46.839277 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:46.839280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:46.840355 (+ 1075us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:46.841248 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.841271 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.842392 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.842395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.843847 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:46.843858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:46.844620 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:46.844624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:46.845061 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:46.845852 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:46.845864 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:46.848370 (+ 2506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:46.851271 (+ 2901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.851274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.851275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.851565 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.851895 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:46.851897 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:46.851899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:46.852034 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:46.852469 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:46.852472 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:46.852595 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 15:07:46.852816 (+ 221us) client_negotiation.cc:240] Negotiation successful
0927 15:07:46.853235 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":384,"spinlock_wait_cycles":195840,"thread_start_us":145,"threads_started":1}
I20260927 15:07:46.856187 14317 heartbeater.cc:347] Connected to a master server at 127.7.17.253:34217
I20260927 15:07:46.856288 14315 heartbeater.cc:347] Connected to a master server at 127.7.17.252:40175
I20260927 15:07:46.856408 14315 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:46.856408 14317 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:46.856913 14316 heartbeater.cc:347] Connected to a master server at 127.7.17.254:33177
I20260927 15:07:46.856986 14316 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:46.857008 14317 heartbeater.cc:511] Master 127.7.17.253:34217 requested a full tablet report, sending...
I20260927 15:07:46.857008 14315 heartbeater.cc:511] Master 127.7.17.252:40175 requested a full tablet report, sending...
I20260927 15:07:46.857134 14316 heartbeater.cc:511] Master 127.7.17.254:33177 requested a full tablet report, sending...
I20260927 15:07:46.858306 13351 ts_manager.cc:194] Re-registered known tserver with Master: 627b253d79314dd39ba841bfedb233e6 (127.7.17.193:45549)
I20260927 15:07:46.858529 13946 ts_manager.cc:194] Registered new tserver with Master: 627b253d79314dd39ba841bfedb233e6 (127.7.17.193:45549)
I20260927 15:07:46.858702 14136 ts_manager.cc:194] Registered new tserver with Master: 627b253d79314dd39ba841bfedb233e6 (127.7.17.193:45549)
I20260927 15:07:46.859028 13351 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:48403
I20260927 15:07:46.959949 13811 heartbeater.cc:503] Master 127.7.17.254:33177 was elected leader, sending a full tablet report...
W20260927 15:07:46.966682 14323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:46.966977 14323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:46.967039 14323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:46.971208 14323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:46.971282 14323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:46.971364 14323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:07:46.976890 14323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.194.keytab
--principal=oryx/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:46871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=41513
--tserver_master_addrs=127.7.17.254:33177,127.7.17.253:34217,127.7.17.252:40175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:46.977980 14323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:46.979038 14323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:46.988746 14329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:46.988741 14328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:46.989015 14323 server_base.cc:1034] running on GCE node
W20260927 15:07:46.989030 14331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:46.989619 14323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:46.990300 14323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:46.991460 14323 hybrid_clock.cc:648] HybridClock initialized: now 1790521666991417 us; error 54 us; skew 500 ppm
Sep 27 15:07:46 dist-test-slave-hhpq krb5kdc[13307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521666, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:46.994210 14323 init.cc:377] Logged in from keytab as oryx/127.7.17.194@KRBTEST.COM (short username oryx)
I20260927 15:07:46.995177 14323 webserver.cc:460] Webserver started at http://127.7.17.194:41513/ using document root <none> and password file <none>
I20260927 15:07:46.995735 14323 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:46.995822 14323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:47.000030 14323 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:07:47.002218 14338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:47.003587 14323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:07:47.003706 14323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d"
format_stamp: "Formatted at 2026-09-27 15:07:45 on dist-test-slave-hhpq"
server_key: "e68592eb19f7a984159aff53125047aa"
server_key_iv: "e33edcff548ea33c41ce61a4e5d5273b"
server_key_version: "encryptionkey@0"
I20260927 15:07:47.004285 14323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:47.037145 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.033410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60074 (local address 127.7.17.252:40175)
0927 15:07:47.033532 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:47.033536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:47.033556 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:07:47.033694 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:47.033696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:47.033756 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:47.033842 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:47.034173 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.034693 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.036028 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.036549 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.036649 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.036777 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:47.036947 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 15:07:47.037023 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 15:07:47.037170 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.033294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:40175 (local address 127.0.0.1:60074)
0927 15:07:47.033463 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:47.033480 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:47.033594 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:47.033864 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:47.033868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:47.033881 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:47.033976 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.033982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.034810 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.034814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.035853 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.035868 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.036686 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.036690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.036789 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.036817 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 15:07:47.036930 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 15:07:47.037037 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 15:07:47.037884 14146 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 15:07:47.038108 14323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:47.038336 14178 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc7b3a82e94e47b28a0f16ceb4f31721. Latest consensus state: current_term: 1 leader_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: NON_VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: true } } }
I20260927 15:07:47.038491 14178 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [sys.catalog]: This master's current role is: LEARNER
I20260927 15:07:47.041621 14323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:47.041812 14323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:47.042433 14323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:47.043335 14323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:47.043382 14323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:47.043422 14323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:47.043450 14323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:47.051988 14323 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:46871
I20260927 15:07:47.052021 14451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:46871 every 8 connection(s)
I20260927 15:07:47.053061 14323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:07:47.054109 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 14323
I20260927 15:07:47.054301 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 13682
I20260927 15:07:47.060312 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:40949
--local_ip_for_outbound_sockets=127.7.17.195
--tserver_master_addrs=127.7.17.254:33177
--webserver_port=37761
--webserver_interface=127.7.17.195
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.195.keytab
--principal=oryx/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.17.254:33177,127.7.17.253:34217
--tserver_master_addrs=127.7.17.254:33177,127.7.17.253:34217,127.7.17.252:40175 with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:07:47 dist-test-slave-hhpq krb5kdc[13307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521666, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM
Sep 27 15:07:47 dist-test-slave-hhpq krb5kdc[13307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521666, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.252@KRBTEST.COM
Sep 27 15:07:47 dist-test-slave-hhpq krb5kdc[13307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521666, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.253@KRBTEST.COM
I20260927 15:07:47.070353 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.055613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:44317 (local address 127.7.17.254:33177)
0927 15:07:47.055752 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:47.055757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:47.057069 (+ 1312us) server_negotiation.cc:407] Connection header received
0927 15:07:47.058169 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:47.058172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:47.058230 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:47.058304 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:47.059508 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.060212 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.061878 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.062239 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.062317 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.065943 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:47.065965 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:47.065969 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:47.066002 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:47.067671 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.068342 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.068346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.068348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.068438 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.068876 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.068880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.068881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.069140 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:47.069223 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:47.070045 (+ 822us) server_negotiation.cc:299] Negotiation successful
0927 15:07:47.070182 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:07:47.070353 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.057613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:44577 (local address 127.7.17.252:40175)
0927 15:07:47.057729 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:47.057734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:47.060106 (+ 2372us) server_negotiation.cc:407] Connection header received
0927 15:07:47.060272 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:47.060276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:47.060325 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:47.060377 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:47.061005 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.061702 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.062929 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.063235 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.063311 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.066374 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:47.066378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:47.066381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:47.066411 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:47.068068 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.068503 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.068507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.068508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.068597 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.068924 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.068928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.068929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.069145 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:47.069219 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:47.070052 (+ 833us) server_negotiation.cc:299] Negotiation successful
0927 15:07:47.070182 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:07:47.071298 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.059461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:44779 (local address 127.7.17.253:34217)
0927 15:07:47.059580 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:47.059584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:47.059598 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:07:47.059640 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:47.059642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:47.059686 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:47.059764 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:47.060257 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.061027 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.062356 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.062675 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.062755 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.067607 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:47.067612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:47.067614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:47.067660 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:47.069414 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.069932 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.069937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.069939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.070031 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.070349 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.070352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.070353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.070573 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:47.070647 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:47.071004 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 15:07:47.071142 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:07:47.072148 14456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.056365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.194:44317)
0927 15:07:47.056908 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:47.056964 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:47.057962 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:47.058490 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:47.058497 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:47.058868 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:47.059359 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.059373 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.060370 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.060373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.061746 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.061755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.062515 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.062518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.062856 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.063639 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:47.063672 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:47.065745 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:47.067863 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:47.067869 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:47.067889 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:47.068228 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:47.068553 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:47.068556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:47.068558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:47.068746 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:47.069291 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:47.069298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:47.069734 (+ 436us) client_negotiation.cc:769] Sending connection context
0927 15:07:47.070913 (+ 1179us) client_negotiation.cc:240] Negotiation successful
0927 15:07:47.071056 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":99,"spinlock_wait_cycles":7040,"thread_start_us":124,"threads_started":1}
I20260927 15:07:47.072165 14458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.056365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:34217 (local address 127.7.17.194:44779)
0927 15:07:47.057790 (+ 1425us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:47.057803 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:47.057970 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:47.059798 (+ 1828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:47.059801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:47.060007 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:47.060117 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.060123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.061408 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.061411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.062234 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.062241 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.062901 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.062904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.062984 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.063677 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:47.063689 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:47.067425 (+ 3736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:47.069563 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:47.069565 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:47.069567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:47.069806 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:47.070155 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:47.070158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:47.070160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:47.070246 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:47.070722 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:47.070725 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:47.070821 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 15:07:47.071501 (+ 680us) client_negotiation.cc:240] Negotiation successful
0927 15:07:47.071652 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1340,"mutex_wait_us":52,"spinlock_wait_cycles":1792,"thread_start_us":131,"threads_started":1}
I20260927 15:07:47.072417 14457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.057508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:40175 (local address 127.7.17.194:44577)
0927 15:07:47.060020 (+ 2512us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:47.060033 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:47.060170 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:47.060515 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:47.060518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:47.060698 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:47.060809 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.060815 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.061849 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.061852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.062794 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.062804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.063483 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.063487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.063577 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.063964 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:47.063976 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:47.066203 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:47.068187 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:47.068190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:47.068192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:47.068398 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:47.068710 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:47.068713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:47.068715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:47.068801 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:47.069291 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:47.069298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:47.069734 (+ 436us) client_negotiation.cc:769] Sending connection context
0927 15:07:47.070598 (+ 864us) client_negotiation.cc:240] Negotiation successful
0927 15:07:47.071003 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2452,"mutex_wait_us":102,"spinlock_wait_cycles":23808,"thread_start_us":76,"threads_started":1}
I20260927 15:07:47.073908 14453 heartbeater.cc:347] Connected to a master server at 127.7.17.254:33177
I20260927 15:07:47.073958 14454 heartbeater.cc:347] Connected to a master server at 127.7.17.253:34217
I20260927 15:07:47.073966 14452 heartbeater.cc:347] Connected to a master server at 127.7.17.252:40175
I20260927 15:07:47.074162 14453 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:47.074162 14454 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:47.074162 14452 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:47.074796 14453 heartbeater.cc:511] Master 127.7.17.254:33177 requested a full tablet report, sending...
I20260927 15:07:47.074796 14454 heartbeater.cc:511] Master 127.7.17.253:34217 requested a full tablet report, sending...
I20260927 15:07:47.074796 14452 heartbeater.cc:511] Master 127.7.17.252:40175 requested a full tablet report, sending...
I20260927 15:07:47.076036 13946 ts_manager.cc:194] Registered new tserver with Master: 3a3d3d46c72546e48fdcdcc8b8996d6d (127.7.17.194:46871)
I20260927 15:07:47.076089 14136 ts_manager.cc:194] Registered new tserver with Master: 3a3d3d46c72546e48fdcdcc8b8996d6d (127.7.17.194:46871)
I20260927 15:07:47.076561 13351 ts_manager.cc:194] Re-registered known tserver with Master: 3a3d3d46c72546e48fdcdcc8b8996d6d (127.7.17.194:46871)
I20260927 15:07:47.077193 13351 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:44317
W20260927 15:07:47.177314 14459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:47.177604 14459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:47.177676 14459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:47.181883 14459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:47.181977 14459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:47.182063 14459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:07:47.187974 14459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.195.keytab
--principal=oryx/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:40949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=37761
--tserver_master_addrs=127.7.17.254:33177,127.7.17.253:34217,127.7.17.252:40175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:47.189251 14459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:47.190529 14459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:47.200544 14465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:47.200539 14467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:47.200533 14464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:47.200780 14459 server_base.cc:1034] running on GCE node
I20260927 15:07:47.201323 14459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:47.201951 14459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:47.203181 14459 hybrid_clock.cc:648] HybridClock initialized: now 1790521667203157 us; error 34 us; skew 500 ppm
Sep 27 15:07:47 dist-test-slave-hhpq krb5kdc[13307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521667, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:47.206041 14459 init.cc:377] Logged in from keytab as oryx/127.7.17.195@KRBTEST.COM (short username oryx)
I20260927 15:07:47.206977 14459 webserver.cc:460] Webserver started at http://127.7.17.195:37761/ using document root <none> and password file <none>
I20260927 15:07:47.207563 14459 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:47.207646 14459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:47.211796 14459 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:07:47.213999 14474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:47.215440 14459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:47.215545 14459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "be770c2746a2448e9afa82fdf76f08ec"
format_stamp: "Formatted at 2026-09-27 15:07:45 on dist-test-slave-hhpq"
server_key: "89c5e459ed11afabb49e73424e88d84c"
server_key_iv: "a8f4288850baf8c11c8b7f641b5ecf11"
server_key_version: "encryptionkey@0"
I20260927 15:07:47.216181 14459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:47.237363 14459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:47.240576 14459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:47.240756 14459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:47.241429 14459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:47.242328 14459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:47.242385 14459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:47.242437 14459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:47.242467 14459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:47.251693 14459 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:40949
I20260927 15:07:47.251748 14587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:40949 every 8 connection(s)
I20260927 15:07:47.252733 14459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:47.258402 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 14459
Sep 27 15:07:47 dist-test-slave-hhpq krb5kdc[13307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521667, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.252@KRBTEST.COM
Sep 27 15:07:47 dist-test-slave-hhpq krb5kdc[13307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521667, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM
Sep 27 15:07:47 dist-test-slave-hhpq krb5kdc[13307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521667, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.253@KRBTEST.COM
I20260927 15:07:47.269948 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.256520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:47917 (local address 127.7.17.254:33177)
0927 15:07:47.256638 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:47.256642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:47.256736 (+ 94us) server_negotiation.cc:407] Connection header received
0927 15:07:47.257414 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:47.257418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:47.257469 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:47.257546 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:47.258915 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.259661 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.261226 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.261545 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.261633 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.265741 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:47.265745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:47.265748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:47.265775 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:47.267271 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.268004 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.268008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.268010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.268106 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.268535 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.268539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.268541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.268760 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:47.268878 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:47.269646 (+ 768us) server_negotiation.cc:299] Negotiation successful
0927 15:07:47.269778 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":48}
I20260927 15:07:47.269948 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.256180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:49741 (local address 127.7.17.252:40175)
0927 15:07:47.256310 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:47.256314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:47.256466 (+ 152us) server_negotiation.cc:407] Connection header received
0927 15:07:47.257182 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:47.257185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:47.257230 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:47.257292 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:47.258592 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.259375 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.261256 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.261569 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.261657 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.265766 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:47.265770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:47.265772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:47.265797 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:47.267378 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.268004 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.268008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.268010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.268106 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.268535 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.268539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.268541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.268749 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:47.268863 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:47.269644 (+ 781us) server_negotiation.cc:299] Negotiation successful
0927 15:07:47.269778 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:07:47.270123 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.254440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:35701 (local address 127.7.17.253:34217)
0927 15:07:47.254591 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:47.254596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:47.255824 (+ 1228us) server_negotiation.cc:407] Connection header received
0927 15:07:47.256941 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:47.256945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:47.257001 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:47.257075 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:47.258662 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.259434 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.262980 (+ 3546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.263324 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.263398 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.265934 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:47.265937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:47.265958 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:47.265986 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:47.267965 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.268418 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.268422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.268424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.268512 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.269163 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.269166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.269168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.269385 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:47.269471 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:47.269836 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 15:07:47.269954 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:07:47.271664 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.256050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:40175 (local address 127.7.17.195:49741)
0927 15:07:47.256383 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:47.256396 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:47.256717 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:47.257325 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:47.257328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:47.257775 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:47.258432 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.258453 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.259533 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.259536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.261108 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.261119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.261822 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.261826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.262088 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.262816 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:47.262839 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:47.265616 (+ 2777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:47.267473 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:47.267476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:47.267478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:47.267894 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:47.268210 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:47.268213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:47.268215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:47.268429 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:47.268918 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:47.268925 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:47.269345 (+ 420us) client_negotiation.cc:769] Sending connection context
0927 15:07:47.270107 (+ 762us) client_negotiation.cc:240] Negotiation successful
0927 15:07:47.270521 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":222,"spinlock_wait_cycles":21120,"thread_start_us":72,"threads_started":1}
I20260927 15:07:47.271922 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.255209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:34217 (local address 127.7.17.195:35701)
0927 15:07:47.255670 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:47.255713 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:47.256709 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:47.257238 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:47.257245 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:47.257775 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:47.258436 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.258456 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.259563 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.259566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.262848 (+ 3282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.262858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.263556 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.263559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.263656 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.264106 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:47.264120 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:47.265793 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:47.268068 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:47.268083 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:47.268086 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:47.268314 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:47.268623 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:47.268626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:47.268628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:47.268775 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:47.269532 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:47.269535 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:47.269632 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 15:07:47.270654 (+ 1022us) client_negotiation.cc:240] Negotiation successful
0927 15:07:47.270800 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":133,"spinlock_wait_cycles":8960,"thread_start_us":108,"threads_started":1}
I20260927 15:07:47.272346 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.256406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.195:47917)
0927 15:07:47.256640 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:47.256652 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:47.256746 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:47.257580 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:47.257583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:47.257778 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:47.258782 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.258790 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.259785 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.259788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.261107 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.261116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.261821 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.261825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.262085 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.262864 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:47.262876 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:47.265575 (+ 2699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:47.267399 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:47.267406 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:47.267426 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:47.267894 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:47.268210 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:47.268213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:47.268215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:47.268429 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:47.268925 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:47.268927 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:47.269345 (+ 418us) client_negotiation.cc:769] Sending connection context
0927 15:07:47.270607 (+ 1262us) client_negotiation.cc:240] Negotiation successful
0927 15:07:47.270733 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":260,"spinlock_wait_cycles":160640,"thread_start_us":91,"threads_started":1}
I20260927 15:07:47.273170 14588 heartbeater.cc:347] Connected to a master server at 127.7.17.252:40175
I20260927 15:07:47.273205 14590 heartbeater.cc:347] Connected to a master server at 127.7.17.253:34217
I20260927 15:07:47.273185 14589 heartbeater.cc:347] Connected to a master server at 127.7.17.254:33177
I20260927 15:07:47.273445 14589 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:47.273590 14588 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:47.273445 14590 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:47.274024 14589 heartbeater.cc:511] Master 127.7.17.254:33177 requested a full tablet report, sending...
I20260927 15:07:47.274201 14590 heartbeater.cc:511] Master 127.7.17.253:34217 requested a full tablet report, sending...
I20260927 15:07:47.274019 14588 heartbeater.cc:511] Master 127.7.17.252:40175 requested a full tablet report, sending...
I20260927 15:07:47.275151 13351 ts_manager.cc:194] Re-registered known tserver with Master: be770c2746a2448e9afa82fdf76f08ec (127.7.17.195:40949)
I20260927 15:07:47.275242 14136 ts_manager.cc:194] Registered new tserver with Master: be770c2746a2448e9afa82fdf76f08ec (127.7.17.195:40949)
I20260927 15:07:47.275836 13946 ts_manager.cc:194] Registered new tserver with Master: be770c2746a2448e9afa82fdf76f08ec (127.7.17.195:40949)
I20260927 15:07:47.275875 13351 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:47917
I20260927 15:07:47.277181 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.260465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44268 (local address 127.7.17.254:33177)
0927 15:07:47.261196 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:47.261200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:47.261210 (+ 10us) server_negotiation.cc:407] Connection header received
0927 15:07:47.261239 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:47.261241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:47.261287 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:47.261357 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:47.262363 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.263580 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.264974 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.265343 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.265440 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.273058 (+ 7618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:47.273062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:47.273065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:47.273092 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:47.274798 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.275359 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.275362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.275364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.275447 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.275729 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.275732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.275734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.275965 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:47.276081 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:47.276887 (+ 806us) server_negotiation.cc:299] Negotiation successful
0927 15:07:47.277022 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":82,"threads_started":1}
I20260927 15:07:47.278432 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.261628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60082 (local address 127.7.17.252:40175)
0927 15:07:47.263972 (+ 2344us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:47.263976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:47.264603 (+ 627us) server_negotiation.cc:407] Connection header received
0927 15:07:47.264643 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:47.264646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:47.264692 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:47.264765 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:47.265648 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.266807 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.267965 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.268287 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.268361 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.270097 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:47.270101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:47.270104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:47.270129 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:47.271816 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.277018 (+ 5202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.277022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.277024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.277132 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.277575 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.277579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.277580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.277794 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:47.277922 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:47.278144 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 15:07:47.278278 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2268,"thread_start_us":93,"threads_started":1}
I20260927 15:07:47.281420 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.260973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47316 (local address 127.7.17.253:34217)
0927 15:07:47.270286 (+ 9313us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:47.270290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:47.270302 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:07:47.270337 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:47.270339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:47.270384 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:47.270453 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:47.272544 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.274231 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.275828 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.276129 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.276196 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.277856 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:47.277861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:47.277864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:47.277889 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:47.279644 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.280064 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.280066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.280068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.280152 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.280467 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.280470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.280472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.280791 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:47.280982 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:47.281155 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 15:07:47.281266 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9252,"thread_start_us":83,"threads_started":1}
I20260927 15:07:47.280725 13351 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:07:47.283948 13351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:47.286634 14607 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: NON_VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: true } } }
I20260927 15:07:47.286800 14607 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:47.291193 13915 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721: attempting to promote NON_VOTER d7a6e46bfaba4e33802c1991a2cc8c0c to VOTER
I20260927 15:07:47.291584 13915 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: false } }
I20260927 15:07:47.292610 14146 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 1 LEARNER]: Refusing update from remote peer bc7b3a82e94e47b28a0f16ceb4f31721: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 15:07:47.293052 13915 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 15:07:47.293388 13957 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 1 FOLLOWER]: Refusing update from remote peer bc7b3a82e94e47b28a0f16ceb4f31721: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 15:07:47.293990 13915 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 15:07:47.295872 14606 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d7a6e46bfaba4e33802c1991a2cc8c0c (127.7.17.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: false } } }
I20260927 15:07:47.296325 13957 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d7a6e46bfaba4e33802c1991a2cc8c0c (127.7.17.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: false } } }
I20260927 15:07:47.296521 14146 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d7a6e46bfaba4e33802c1991a2cc8c0c (127.7.17.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: false } } }
I20260927 15:07:47.297523 14606 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: false } } }
I20260927 15:07:47.297636 14606 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:47.297762 14606 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: false } } }
I20260927 15:07:47.297844 14606 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:47.297942 14606 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: false } } }
I20260927 15:07:47.298018 14606 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:47.298266 14615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: false } } }
I20260927 15:07:47.298399 14615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:07:47.298942 14178 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: false } } }
I20260927 15:07:47.299075 14178 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:07:47.305641 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.299690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:46871 (local address 127.0.0.1:35444)
0927 15:07:47.299859 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:47.299875 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:47.299956 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:47.301797 (+ 1841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:47.301800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:47.301817 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:47.301902 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.301908 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.303324 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.303328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.304106 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.304115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.305243 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.305247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.305337 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.305362 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:07:47.305423 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 15:07:47.305497 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":37}
I20260927 15:07:47.305694 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.299865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35444 (local address 127.7.17.194:46871)
0927 15:07:47.301369 (+ 1504us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:47.301383 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:47.301404 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:07:47.301476 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:47.301480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:47.301648 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:47.302040 (+ 392us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:47.302193 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.303227 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.304309 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.305148 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.305249 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.305403 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:47.305458 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 15:07:47.305551 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1402,"thread_start_us":119,"threads_started":1}
I20260927 15:07:47.305989 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.300647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:45549 (local address 127.0.0.1:39656)
0927 15:07:47.300984 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:47.300996 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:47.301075 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:47.301564 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:47.301567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:47.301580 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:47.301649 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.301654 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.302897 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.302900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.304106 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.304116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.305615 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.305618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.305701 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.305728 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 15:07:47.305775 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 15:07:47.305849 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":2,"thread_start_us":75,"threads_started":1}
I20260927 15:07:47.306275 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.300184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:40949 (local address 127.0.0.1:33202)
0927 15:07:47.300418 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:47.300430 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:47.300505 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:47.301468 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:47.301472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:47.301490 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:47.301636 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.301643 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.303114 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.303117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.304254 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.304261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.305891 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.305894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.305976 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.305999 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:07:47.306081 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 15:07:47.306151 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":36,"thread_start_us":104,"threads_started":1}
I20260927 15:07:47.306682 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.300559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33202 (local address 127.7.17.195:40949)
0927 15:07:47.300913 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:47.300925 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:47.300952 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:07:47.301040 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:47.301047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:47.301315 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:47.301677 (+ 362us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:47.301829 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.303003 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.304353 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.305144 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.305249 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.305403 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:47.306440 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0927 15:07:47.306550 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":104,"threads_started":1}
I20260927 15:07:47.306767 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.300913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39656 (local address 127.7.17.193:45549)
0927 15:07:47.301170 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:47.301184 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:47.301205 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:07:47.301266 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:47.301270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:47.301419 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:47.301791 (+ 372us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:47.301927 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.302792 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.304237 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.305486 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.306231 (+ 745us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.306500 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:47.306556 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 15:07:47.306643 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":58,"threads_started":1}
I20260927 15:07:47.309188 14386 tablet_service.cc:1467] Processing CreateTablet for tablet 5225101713754371af21afb57e588b6c (DEFAULT_TABLE table=test-table [id=e214a7fb5b134c4f917102885e95c771]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:47.309535 14249 tablet_service.cc:1467] Processing CreateTablet for tablet 5225101713754371af21afb57e588b6c (DEFAULT_TABLE table=test-table [id=e214a7fb5b134c4f917102885e95c771]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:47.310365 14386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5225101713754371af21afb57e588b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:47.310678 14249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5225101713754371af21afb57e588b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:47.314895 14519 tablet_service.cc:1467] Processing CreateTablet for tablet 5225101713754371af21afb57e588b6c (DEFAULT_TABLE table=test-table [id=e214a7fb5b134c4f917102885e95c771]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:47.315963 14519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5225101713754371af21afb57e588b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:47.318706 14621 tablet_bootstrap.cc:492] T 5225101713754371af21afb57e588b6c P 3a3d3d46c72546e48fdcdcc8b8996d6d: Bootstrap starting.
I20260927 15:07:47.319170 14622 tablet_bootstrap.cc:492] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6: Bootstrap starting.
I20260927 15:07:47.322327 14621 tablet_bootstrap.cc:654] T 5225101713754371af21afb57e588b6c P 3a3d3d46c72546e48fdcdcc8b8996d6d: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:47.322916 14622 tablet_bootstrap.cc:654] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:47.323257 14621 log.cc:824] T 5225101713754371af21afb57e588b6c P 3a3d3d46c72546e48fdcdcc8b8996d6d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:47.323622 14622 log.cc:824] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:47.324615 14625 tablet_bootstrap.cc:492] T 5225101713754371af21afb57e588b6c P be770c2746a2448e9afa82fdf76f08ec: Bootstrap starting.
I20260927 15:07:47.325894 14622 tablet_bootstrap.cc:492] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6: No bootstrap required, opened a new log
I20260927 15:07:47.326097 14622 ts_tablet_manager.cc:1397] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 15:07:47.326429 14621 tablet_bootstrap.cc:492] T 5225101713754371af21afb57e588b6c P 3a3d3d46c72546e48fdcdcc8b8996d6d: No bootstrap required, opened a new log
I20260927 15:07:47.326635 14621 ts_tablet_manager.cc:1397] T 5225101713754371af21afb57e588b6c P 3a3d3d46c72546e48fdcdcc8b8996d6d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260927 15:07:47.327189 14625 tablet_bootstrap.cc:654] T 5225101713754371af21afb57e588b6c P be770c2746a2448e9afa82fdf76f08ec: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:47.327994 14625 log.cc:824] T 5225101713754371af21afb57e588b6c P be770c2746a2448e9afa82fdf76f08ec: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:47.329881 14621 raft_consensus.cc:348] T 5225101713754371af21afb57e588b6c P 3a3d3d46c72546e48fdcdcc8b8996d6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 46871 } } peers { permanent_uuid: "be770c2746a2448e9afa82fdf76f08ec" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 40949 } } peers { permanent_uuid: "627b253d79314dd39ba841bfedb233e6" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 45549 } }
I20260927 15:07:47.330106 14621 raft_consensus.cc:374] T 5225101713754371af21afb57e588b6c P 3a3d3d46c72546e48fdcdcc8b8996d6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:47.330158 14621 raft_consensus.cc:729] T 5225101713754371af21afb57e588b6c P 3a3d3d46c72546e48fdcdcc8b8996d6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3d3d46c72546e48fdcdcc8b8996d6d, State: Initialized, Role: FOLLOWER
I20260927 15:07:47.330170 14625 tablet_bootstrap.cc:492] T 5225101713754371af21afb57e588b6c P be770c2746a2448e9afa82fdf76f08ec: No bootstrap required, opened a new log
I20260927 15:07:47.330344 14625 ts_tablet_manager.cc:1397] T 5225101713754371af21afb57e588b6c P be770c2746a2448e9afa82fdf76f08ec: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:07:47.330590 14621 consensus_queue.cc:260] T 5225101713754371af21afb57e588b6c P 3a3d3d46c72546e48fdcdcc8b8996d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 46871 } } peers { permanent_uuid: "be770c2746a2448e9afa82fdf76f08ec" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 40949 } } peers { permanent_uuid: "627b253d79314dd39ba841bfedb233e6" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 45549 } }
I20260927 15:07:47.330474 14622 raft_consensus.cc:348] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 46871 } } peers { permanent_uuid: "be770c2746a2448e9afa82fdf76f08ec" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 40949 } } peers { permanent_uuid: "627b253d79314dd39ba841bfedb233e6" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 45549 } }
I20260927 15:07:47.330725 14622 raft_consensus.cc:374] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:47.330781 14622 raft_consensus.cc:729] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 627b253d79314dd39ba841bfedb233e6, State: Initialized, Role: FOLLOWER
I20260927 15:07:47.331197 14622 consensus_queue.cc:260] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 46871 } } peers { permanent_uuid: "be770c2746a2448e9afa82fdf76f08ec" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 40949 } } peers { permanent_uuid: "627b253d79314dd39ba841bfedb233e6" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 45549 } }
I20260927 15:07:47.331471 14621 ts_tablet_manager.cc:1428] T 5225101713754371af21afb57e588b6c P 3a3d3d46c72546e48fdcdcc8b8996d6d: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260927 15:07:47.332113 14622 ts_tablet_manager.cc:1428] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260927 15:07:47.332659 14316 heartbeater.cc:503] Master 127.7.17.254:33177 was elected leader, sending a full tablet report...
I20260927 15:07:47.333129 14453 heartbeater.cc:503] Master 127.7.17.254:33177 was elected leader, sending a full tablet report...
W20260927 15:07:47.336335 14318 tablet.cc:2369] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:07:47.338827 14625 raft_consensus.cc:348] T 5225101713754371af21afb57e588b6c P be770c2746a2448e9afa82fdf76f08ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 46871 } } peers { permanent_uuid: "be770c2746a2448e9afa82fdf76f08ec" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 40949 } } peers { permanent_uuid: "627b253d79314dd39ba841bfedb233e6" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 45549 } }
I20260927 15:07:47.339069 14625 raft_consensus.cc:374] T 5225101713754371af21afb57e588b6c P be770c2746a2448e9afa82fdf76f08ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:47.339112 14625 raft_consensus.cc:729] T 5225101713754371af21afb57e588b6c P be770c2746a2448e9afa82fdf76f08ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be770c2746a2448e9afa82fdf76f08ec, State: Initialized, Role: FOLLOWER
I20260927 15:07:47.339538 14625 consensus_queue.cc:260] T 5225101713754371af21afb57e588b6c P be770c2746a2448e9afa82fdf76f08ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 46871 } } peers { permanent_uuid: "be770c2746a2448e9afa82fdf76f08ec" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 40949 } } peers { permanent_uuid: "627b253d79314dd39ba841bfedb233e6" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 45549 } }
I20260927 15:07:47.340188 14589 heartbeater.cc:503] Master 127.7.17.254:33177 was elected leader, sending a full tablet report...
I20260927 15:07:47.340605 14625 ts_tablet_manager.cc:1428] T 5225101713754371af21afb57e588b6c P be770c2746a2448e9afa82fdf76f08ec: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260927 15:07:47.396242 14628 raft_consensus.cc:482] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:47.396468 14628 raft_consensus.cc:504] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 46871 } } peers { permanent_uuid: "be770c2746a2448e9afa82fdf76f08ec" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 40949 } } peers { permanent_uuid: "627b253d79314dd39ba841bfedb233e6" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 45549 } }
I20260927 15:07:47.397887 14628 leader_election.cc:288] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a3d3d46c72546e48fdcdcc8b8996d6d (127.7.17.194:46871), be770c2746a2448e9afa82fdf76f08ec (127.7.17.195:40949)
I20260927 15:07:47.402362 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.398118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:40949 (local address 127.7.17.193:51019)
0927 15:07:47.398562 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:47.398577 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:47.398710 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:47.399294 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:47.399297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:47.399314 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:47.399412 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.399418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.400262 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.400265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.401089 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.401098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.401896 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.401900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.401990 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.402016 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:07:47.402126 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 15:07:47.402202 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":1,"spinlock_wait_cycles":6144,"thread_start_us":90,"threads_started":1}
I20260927 15:07:47.402372 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.398274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:51019 (local address 127.7.17.195:40949)
0927 15:07:47.398389 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:47.398393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:47.398672 (+ 279us) server_negotiation.cc:407] Connection header received
0927 15:07:47.399092 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:47.399095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:47.399142 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:47.399202 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:47.399521 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.400143 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.401206 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.401769 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.401903 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.401957 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:47.402172 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 15:07:47.402253 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:07:47.402415 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.398306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:46533 (local address 127.7.17.194:46871)
0927 15:07:47.398489 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:47.398493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:47.398703 (+ 210us) server_negotiation.cc:407] Connection header received
0927 15:07:47.398836 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:47.398839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:47.398892 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:47.398961 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:47.399208 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.399803 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.401337 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.401926 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.402024 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.402066 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:47.402250 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 15:07:47.402316 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 15:07:47.402751 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.398118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:46871 (local address 127.7.17.193:46533)
0927 15:07:47.398563 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:47.398577 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:47.398710 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:47.398984 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:47.398986 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:47.399001 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:47.399098 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.399102 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.399907 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.399910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.401214 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:47.401225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.402028 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.402031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:47.402122 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.402143 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:07:47.402565 (+ 422us) client_negotiation.cc:240] Negotiation successful
0927 15:07:47.402632 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":255,"spinlock_wait_cycles":5120,"thread_start_us":49,"threads_started":1}
I20260927 15:07:47.403349 14533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5225101713754371af21afb57e588b6c" candidate_uuid: "627b253d79314dd39ba841bfedb233e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be770c2746a2448e9afa82fdf76f08ec" is_pre_election: true
I20260927 15:07:47.403581 14406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5225101713754371af21afb57e588b6c" candidate_uuid: "627b253d79314dd39ba841bfedb233e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" is_pre_election: true
I20260927 15:07:47.403635 14533 raft_consensus.cc:2454] T 5225101713754371af21afb57e588b6c P be770c2746a2448e9afa82fdf76f08ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 627b253d79314dd39ba841bfedb233e6 in term 0.
I20260927 15:07:47.403852 14406 raft_consensus.cc:2454] T 5225101713754371af21afb57e588b6c P 3a3d3d46c72546e48fdcdcc8b8996d6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 627b253d79314dd39ba841bfedb233e6 in term 0.
I20260927 15:07:47.404086 14204 leader_election.cc:302] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 627b253d79314dd39ba841bfedb233e6, be770c2746a2448e9afa82fdf76f08ec; no voters:
I20260927 15:07:47.404364 14628 raft_consensus.cc:2781] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:47.404448 14628 raft_consensus.cc:482] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:47.404495 14628 raft_consensus.cc:3037] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:47.405519 14628 raft_consensus.cc:504] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 46871 } } peers { permanent_uuid: "be770c2746a2448e9afa82fdf76f08ec" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 40949 } } peers { permanent_uuid: "627b253d79314dd39ba841bfedb233e6" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 45549 } }
I20260927 15:07:47.405845 14628 leader_election.cc:288] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6 [CANDIDATE]: Term 1 election: Requested vote from peers 3a3d3d46c72546e48fdcdcc8b8996d6d (127.7.17.194:46871), be770c2746a2448e9afa82fdf76f08ec (127.7.17.195:40949)
I20260927 15:07:47.406211 14406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5225101713754371af21afb57e588b6c" candidate_uuid: "627b253d79314dd39ba841bfedb233e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d"
I20260927 15:07:47.406311 14533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5225101713754371af21afb57e588b6c" candidate_uuid: "627b253d79314dd39ba841bfedb233e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be770c2746a2448e9afa82fdf76f08ec"
I20260927 15:07:47.406340 14406 raft_consensus.cc:3037] T 5225101713754371af21afb57e588b6c P 3a3d3d46c72546e48fdcdcc8b8996d6d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:47.406430 14533 raft_consensus.cc:3037] T 5225101713754371af21afb57e588b6c P be770c2746a2448e9afa82fdf76f08ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:47.407361 14406 raft_consensus.cc:2454] T 5225101713754371af21afb57e588b6c P 3a3d3d46c72546e48fdcdcc8b8996d6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 627b253d79314dd39ba841bfedb233e6 in term 1.
I20260927 15:07:47.407526 14533 raft_consensus.cc:2454] T 5225101713754371af21afb57e588b6c P be770c2746a2448e9afa82fdf76f08ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 627b253d79314dd39ba841bfedb233e6 in term 1.
I20260927 15:07:47.407651 14203 leader_election.cc:302] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3d3d46c72546e48fdcdcc8b8996d6d, 627b253d79314dd39ba841bfedb233e6; no voters:
I20260927 15:07:47.407835 14628 raft_consensus.cc:2781] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:47.408032 14628 raft_consensus.cc:686] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6 [term 1 LEADER]: Becoming Leader. State: Replica: 627b253d79314dd39ba841bfedb233e6, State: Running, Role: LEADER
I20260927 15:07:47.408401 14628 consensus_queue.cc:237] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 46871 } } peers { permanent_uuid: "be770c2746a2448e9afa82fdf76f08ec" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 40949 } } peers { permanent_uuid: "627b253d79314dd39ba841bfedb233e6" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 45549 } }
I20260927 15:07:47.411815 13352 catalog_manager.cc:5495] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 627b253d79314dd39ba841bfedb233e6 (127.7.17.193). New cstate: current_term: 1 leader_uuid: "627b253d79314dd39ba841bfedb233e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 46871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be770c2746a2448e9afa82fdf76f08ec" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 40949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "627b253d79314dd39ba841bfedb233e6" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 45549 } health_report { overall_health: HEALTHY } } }
I20260927 15:07:47.426064 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.422597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39670 (local address 127.7.17.193:45549)
0927 15:07:47.422711 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:47.422714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:47.422726 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:07:47.422795 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:47.422797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:47.422850 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:47.422933 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:47.423151 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.423637 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.424528 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.424754 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.424867 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.425271 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:47.425702 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:47.425799 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:47.425862 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 15:07:47.425913 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:07:47.432132 14406 raft_consensus.cc:1261] T 5225101713754371af21afb57e588b6c P 3a3d3d46c72546e48fdcdcc8b8996d6d [term 1 FOLLOWER]: Refusing update from remote peer 627b253d79314dd39ba841bfedb233e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:47.432132 14533 raft_consensus.cc:1261] T 5225101713754371af21afb57e588b6c P be770c2746a2448e9afa82fdf76f08ec [term 1 FOLLOWER]: Refusing update from remote peer 627b253d79314dd39ba841bfedb233e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:47.432889 14632 consensus_queue.cc:1035] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 46871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:47.433095 14628 consensus_queue.cc:1035] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be770c2746a2448e9afa82fdf76f08ec" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 40949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:47.442237 14637 mvcc.cc:205] Tried to move back new op lower bound from 7333976749794336768 to 7333976749707517952. Current Snapshot: MvccSnapshot[applied={T|T < 7333976749794336768}]
I20260927 15:07:47.452105 13352 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:47.452315 13352 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:47.457077 13352 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc7b3a82e94e47b28a0f16ceb4f31721: Sending DeleteTablet for 3 replicas of tablet 5225101713754371af21afb57e588b6c
I20260927 15:07:47.457887 14519 tablet_service.cc:1514] Processing DeleteTablet for tablet 5225101713754371af21afb57e588b6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:47 UTC) from {username='oryx', principal='oryx/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:33202
I20260927 15:07:47.457995 14386 tablet_service.cc:1514] Processing DeleteTablet for tablet 5225101713754371af21afb57e588b6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:47 UTC) from {username='oryx', principal='oryx/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:35444
I20260927 15:07:47.458596 14249 tablet_service.cc:1514] Processing DeleteTablet for tablet 5225101713754371af21afb57e588b6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:47 UTC) from {username='oryx', principal='oryx/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:39656
I20260927 15:07:47.459045 14646 tablet_replica.cc:330] T 5225101713754371af21afb57e588b6c P 3a3d3d46c72546e48fdcdcc8b8996d6d: stopping tablet replica
I20260927 15:07:47.459472 14646 raft_consensus.cc:2229] T 5225101713754371af21afb57e588b6c P 3a3d3d46c72546e48fdcdcc8b8996d6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:47.459824 14646 raft_consensus.cc:2258] T 5225101713754371af21afb57e588b6c P 3a3d3d46c72546e48fdcdcc8b8996d6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:47.461514 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 13320
I20260927 15:07:47.461752 14646 ts_tablet_manager.cc:1905] T 5225101713754371af21afb57e588b6c P 3a3d3d46c72546e48fdcdcc8b8996d6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:47.464551 14646 ts_tablet_manager.cc:1918] T 5225101713754371af21afb57e588b6c P 3a3d3d46c72546e48fdcdcc8b8996d6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:47.464653 14646 log.cc:1196] T 5225101713754371af21afb57e588b6c P 3a3d3d46c72546e48fdcdcc8b8996d6d: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-1/wal/wals/5225101713754371af21afb57e588b6c
I20260927 15:07:47.464982 14646 ts_tablet_manager.cc:1939] T 5225101713754371af21afb57e588b6c P 3a3d3d46c72546e48fdcdcc8b8996d6d: Deleting consensus metadata
W20260927 15:07:47.468586 14340 connection.cc:535] server connection from 127.0.0.1:35444 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 15:07:47.469288 14645 tablet_replica.cc:330] T 5225101713754371af21afb57e588b6c P be770c2746a2448e9afa82fdf76f08ec: stopping tablet replica
I20260927 15:07:47.469328 14647 tablet_replica.cc:330] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6: stopping tablet replica
I20260927 15:07:47.469681 14645 raft_consensus.cc:2229] T 5225101713754371af21afb57e588b6c P be770c2746a2448e9afa82fdf76f08ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:47.469822 14647 raft_consensus.cc:2229] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:07:47.470085 14645 raft_consensus.cc:2258] T 5225101713754371af21afb57e588b6c P be770c2746a2448e9afa82fdf76f08ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:47.470379 14647 raft_consensus.cc:2258] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:47.470398 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.459887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47324 (local address 127.7.17.253:34217)
0927 15:07:47.459990 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:47.459994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:47.460007 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:07:47.460045 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:47.460048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:47.460093 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:47.460160 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:47.461092 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.461864 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.463580 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.463923 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.463994 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.466381 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:47.466386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:47.466388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:47.466417 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:47.468007 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.468491 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.468495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.468496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.468600 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.468949 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.468952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.468954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.469215 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:47.469323 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:47.470103 (+ 780us) server_negotiation.cc:299] Negotiation successful
0927 15:07:47.470242 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 15:07:47.473577 14647 ts_tablet_manager.cc:1905] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:47.476413 14647 ts_tablet_manager.cc:1918] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:47.476536 14647 log.cc:1196] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-0/wal/wals/5225101713754371af21afb57e588b6c
I20260927 15:07:47.476883 14647 ts_tablet_manager.cc:1939] T 5225101713754371af21afb57e588b6c P 627b253d79314dd39ba841bfedb233e6: Deleting consensus metadata
I20260927 15:07:47.477948 14645 ts_tablet_manager.cc:1905] T 5225101713754371af21afb57e588b6c P be770c2746a2448e9afa82fdf76f08ec: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 15:07:47.478017 14205 connection.cc:441] server connection from 127.0.0.1:39656 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39656 (request call id 1) could send its response
I20260927 15:07:47.480700 14645 ts_tablet_manager.cc:1918] T 5225101713754371af21afb57e588b6c P be770c2746a2448e9afa82fdf76f08ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:47.480803 14645 log.cc:1196] T 5225101713754371af21afb57e588b6c P be770c2746a2448e9afa82fdf76f08ec: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/5225101713754371af21afb57e588b6c
I20260927 15:07:47.481137 14645 ts_tablet_manager.cc:1939] T 5225101713754371af21afb57e588b6c P be770c2746a2448e9afa82fdf76f08ec: Deleting consensus metadata
I20260927 15:07:47.482093 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.471876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47330 (local address 127.7.17.253:34217)
0927 15:07:47.472446 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:47.472451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:47.472544 (+ 93us) server_negotiation.cc:407] Connection header received
0927 15:07:47.472736 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:47.472739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:47.472787 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:47.472853 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:47.473817 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.474618 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.475930 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.476253 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.476326 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.478227 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:47.478231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:47.478234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:47.478262 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:47.479913 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.480422 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.480426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.480428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.480513 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.480879 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.480883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.480885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.481107 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:47.481249 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:47.481811 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 15:07:47.481932 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498}
W20260927 15:07:47.482434 14476 connection.cc:441] server connection from 127.0.0.1:33202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33202 (request call id 1) could send its response
I20260927 15:07:47.483179 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.472358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60086 (local address 127.7.17.252:40175)
0927 15:07:47.472468 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:47.472472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:47.472808 (+ 336us) server_negotiation.cc:407] Connection header received
0927 15:07:47.473007 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:47.473010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:47.473061 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:47.473129 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:47.474037 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.474776 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.475907 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.476253 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.476326 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.479201 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:47.479205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:47.479208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:47.479237 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:47.480776 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.481443 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.481447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.481449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.481533 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.481931 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.481934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.481936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.482159 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:47.482276 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:47.482905 (+ 629us) server_negotiation.cc:299] Negotiation successful
0927 15:07:47.483025 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:07:47.487077 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:47.459738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60084 (local address 127.7.17.252:40175)
0927 15:07:47.459862 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:47.459866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:47.478409 (+ 18543us) server_negotiation.cc:407] Connection header received
0927 15:07:47.478673 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:47.478677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:47.478740 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:47.478824 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:47.479461 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.480172 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.481281 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:47.481595 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:47.481691 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:47.483128 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:47.483132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:47.483134 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:47.483155 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:47.484710 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.485245 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.485248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.485249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.485320 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:47.485622 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:47.485626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:47.485628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:47.485834 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:47.485945 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:47.486866 (+ 921us) server_negotiation.cc:299] Negotiation successful
0927 15:07:47.486966 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:07:48.444489 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:48.443989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.195:50933)
0927 15:07:48.444271 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:48.444371 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":92,"threads_started":1}
W20260927 15:07:48.444864 14589 heartbeater.cc:644] Failed to heartbeat to 127.7.17.254:33177 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
I20260927 15:07:48.447553 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:48.447061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.194:53353)
0927 15:07:48.447352 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:48.447443 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":78,"threads_started":1}
W20260927 15:07:48.448038 14453 heartbeater.cc:644] Failed to heartbeat to 127.7.17.254:33177 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
I20260927 15:07:48.448375 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:48.447995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.193:59871)
0927 15:07:48.448210 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:48.448283 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
W20260927 15:07:48.448691 14316 heartbeater.cc:644] Failed to heartbeat to 127.7.17.254:33177 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
I20260927 15:07:48.957753 14658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bc7b3a82e94e47b28a0f16ceb4f31721)
I20260927 15:07:48.957826 14659 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bc7b3a82e94e47b28a0f16ceb4f31721)
I20260927 15:07:48.957890 14658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: false } }
I20260927 15:07:48.958148 14659 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: false } }
I20260927 15:07:48.958463 14658 leader_election.cc:288] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc7b3a82e94e47b28a0f16ceb4f31721 (127.7.17.254:33177), d7a6e46bfaba4e33802c1991a2cc8c0c (127.7.17.252:40175)
I20260927 15:07:48.959240 14660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:48.958683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.0.0.1:44726)
0927 15:07:48.958996 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:48.959087 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":60,"threads_started":1}
W20260927 15:07:48.959422 13939 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
I20260927 15:07:48.959661 14659 leader_election.cc:288] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc7b3a82e94e47b28a0f16ceb4f31721 (127.7.17.254:33177), 3c1aa5b16f0847f2ac9bb445628f78f7 (127.7.17.253:34217)
I20260927 15:07:48.959897 14663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:48.959412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.0.0.1:44728)
0927 15:07:48.959737 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:48.959810 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":52,"threads_started":1}
W20260927 15:07:48.960062 14121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
I20260927 15:07:48.960299 14660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:48.960045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.0.0.1:44734)
0927 15:07:48.960187 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:48.960233 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260927 15:07:48.960428 13939 leader_election.cc:334] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bc7b3a82e94e47b28a0f16ceb4f31721 (127.7.17.254:33177): Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
I20260927 15:07:48.960932 14663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:48.960704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.0.0.1:44736)
0927 15:07:48.960805 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:48.960874 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 15:07:48.961099 14121 leader_election.cc:334] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bc7b3a82e94e47b28a0f16ceb4f31721 (127.7.17.254:33177): Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
I20260927 15:07:48.964311 14662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:48.958861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60102 (local address 127.7.17.252:40175)
0927 15:07:48.959097 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:48.959101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:48.959200 (+ 99us) server_negotiation.cc:407] Connection header received
0927 15:07:48.960083 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:48.960086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:48.960130 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:48.960266 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:48.960726 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:48.961513 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:48.963097 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:48.963732 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:48.963845 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:48.963906 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:48.964088 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 15:07:48.964164 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":51,"threads_started":1}
I20260927 15:07:48.964313 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:48.959482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:34217 (local address 127.0.0.1:47344)
0927 15:07:48.959847 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:48.959860 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:48.959949 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:48.960360 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:48.960363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:48.960376 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:48.960471 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:48.960478 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:48.961642 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:48.961645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:48.962944 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:48.962955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:48.963863 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:48.963866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:48.963963 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:48.963987 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:07:48.964065 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 15:07:48.964144 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":67,"threads_started":1}
I20260927 15:07:48.964349 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:48.959688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47344 (local address 127.7.17.253:34217)
0927 15:07:48.960154 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:48.960163 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:48.960175 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:07:48.960214 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:48.960217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:48.960258 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:48.960568 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:48.960639 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:48.961509 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:48.963102 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:48.963740 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:48.963845 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:48.963906 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:48.964126 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 15:07:48.964224 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":405,"thread_start_us":86,"threads_started":1}
I20260927 15:07:48.964360 14661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:48.958688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:40175 (local address 127.0.0.1:60102)
0927 15:07:48.959101 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:48.959117 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:48.959242 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:48.960431 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:48.960434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:48.960449 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:48.960605 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:48.960613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:48.961621 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:48.961625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:48.962944 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:48.962955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:48.963863 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:48.963866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:48.963963 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:48.963987 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:07:48.964065 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 15:07:48.964207 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":52,"thread_start_us":55,"threads_started":1}
I20260927 15:07:48.964811 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" is_pre_election: true
I20260927 15:07:48.964920 14146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" is_pre_election: true
I20260927 15:07:48.965049 13957 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7a6e46bfaba4e33802c1991a2cc8c0c in term 1.
I20260927 15:07:48.965112 14146 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c1aa5b16f0847f2ac9bb445628f78f7 in term 1.
I20260927 15:07:48.965385 14123 leader_election.cc:302] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c1aa5b16f0847f2ac9bb445628f78f7, d7a6e46bfaba4e33802c1991a2cc8c0c; no voters: bc7b3a82e94e47b28a0f16ceb4f31721
I20260927 15:07:48.965384 13941 leader_election.cc:302] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c1aa5b16f0847f2ac9bb445628f78f7, d7a6e46bfaba4e33802c1991a2cc8c0c; no voters: bc7b3a82e94e47b28a0f16ceb4f31721
I20260927 15:07:48.965567 14658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 15:07:48.965628 14658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bc7b3a82e94e47b28a0f16ceb4f31721)
I20260927 15:07:48.965675 14658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:07:48.965749 14659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 15:07:48.965808 14659 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader bc7b3a82e94e47b28a0f16ceb4f31721)
I20260927 15:07:48.965847 14659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:07:48.966728 14658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: false } }
I20260927 15:07:48.966754 14659 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: false } }
I20260927 15:07:48.967057 14658 leader_election.cc:288] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [CANDIDATE]: Term 2 election: Requested vote from peers bc7b3a82e94e47b28a0f16ceb4f31721 (127.7.17.254:33177), d7a6e46bfaba4e33802c1991a2cc8c0c (127.7.17.252:40175)
I20260927 15:07:48.967090 14659 leader_election.cc:288] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [CANDIDATE]: Term 2 election: Requested vote from peers bc7b3a82e94e47b28a0f16ceb4f31721 (127.7.17.254:33177), 3c1aa5b16f0847f2ac9bb445628f78f7 (127.7.17.253:34217)
I20260927 15:07:48.967461 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:48.967207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.0.0.1:44750)
0927 15:07:48.967312 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:48.967365 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 15:07:48.967438 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7"
I20260927 15:07:48.967536 14146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c"
I20260927 15:07:48.967590 13957 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d7a6e46bfaba4e33802c1991a2cc8c0c in current term 2: Already voted for candidate 3c1aa5b16f0847f2ac9bb445628f78f7 in this term.
I20260927 15:07:48.967670 14146 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c1aa5b16f0847f2ac9bb445628f78f7 in current term 2: Already voted for candidate d7a6e46bfaba4e33802c1991a2cc8c0c in this term.
I20260927 15:07:48.967741 14661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:48.967209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.0.0.1:44740)
0927 15:07:48.967620 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:48.967670 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345}
I20260927 15:07:48.968246 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:48.968058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.0.0.1:44758)
0927 15:07:48.968146 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:48.968193 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 15:07:48.968340 14121 leader_election.cc:334] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bc7b3a82e94e47b28a0f16ceb4f31721 (127.7.17.254:33177): Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
I20260927 15:07:48.968367 14661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:48.968153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.0.0.1:44770)
0927 15:07:48.968259 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:48.968306 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 15:07:48.968420 14121 leader_election.cc:302] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7a6e46bfaba4e33802c1991a2cc8c0c; no voters: 3c1aa5b16f0847f2ac9bb445628f78f7, bc7b3a82e94e47b28a0f16ceb4f31721
I20260927 15:07:48.968537 14659 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260927 15:07:48.968657 13939 leader_election.cc:334] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bc7b3a82e94e47b28a0f16ceb4f31721 (127.7.17.254:33177): Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
I20260927 15:07:48.968746 13939 leader_election.cc:302] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c1aa5b16f0847f2ac9bb445628f78f7; no voters: bc7b3a82e94e47b28a0f16ceb4f31721, d7a6e46bfaba4e33802c1991a2cc8c0c
I20260927 15:07:48.968874 14658 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 15:07:49.446206 14668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:49.445625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.195:46423)
0927 15:07:49.445945 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:49.446024 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":100,"threads_started":1}
I20260927 15:07:49.448980 14669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:49.448541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.194:41207)
0927 15:07:49.448788 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:49.448878 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":88,"threads_started":1}
I20260927 15:07:49.449671 14670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:49.449269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.193:46715)
0927 15:07:49.449501 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:49.449557 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260927 15:07:50.447556 14671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.447090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.195:41085)
0927 15:07:50.447364 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.447438 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":86,"threads_started":1}
I20260927 15:07:50.450302 14672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.449830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.194:52377)
0927 15:07:50.450134 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.450199 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":84,"threads_started":1}
I20260927 15:07:50.450917 14673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.450307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.193:32789)
0927 15:07:50.450587 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.450824 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":60,"threads_started":1}
I20260927 15:07:50.567301 14674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:50.567440 14674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: false } }
I20260927 15:07:50.567864 14674 leader_election.cc:288] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bc7b3a82e94e47b28a0f16ceb4f31721 (127.7.17.254:33177), d7a6e46bfaba4e33802c1991a2cc8c0c (127.7.17.252:40175)
I20260927 15:07:50.568370 14146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" is_pre_election: true
I20260927 15:07:50.568490 14675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.568069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.0.0.1:44968)
0927 15:07:50.568311 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.568387 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":69,"threads_started":1}
I20260927 15:07:50.568540 14146 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c1aa5b16f0847f2ac9bb445628f78f7 in term 2.
I20260927 15:07:50.568919 13941 leader_election.cc:302] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c1aa5b16f0847f2ac9bb445628f78f7, d7a6e46bfaba4e33802c1991a2cc8c0c; no voters:
I20260927 15:07:50.569136 14674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 15:07:50.569193 14674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:50.569221 14674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 2 FOLLOWER]: Advancing to term 3
I20260927 15:07:50.569511 14675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.569264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.0.0.1:44982)
0927 15:07:50.569363 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.569439 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 15:07:50.569692 13939 leader_election.cc:334] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bc7b3a82e94e47b28a0f16ceb4f31721 (127.7.17.254:33177): Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
I20260927 15:07:50.570360 14674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: false } }
I20260927 15:07:50.570688 14674 leader_election.cc:288] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [CANDIDATE]: Term 3 election: Requested vote from peers bc7b3a82e94e47b28a0f16ceb4f31721 (127.7.17.254:33177), d7a6e46bfaba4e33802c1991a2cc8c0c (127.7.17.252:40175)
I20260927 15:07:50.571038 14675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.570834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.0.0.1:44990)
0927 15:07:50.570926 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.570973 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 15:07:50.571059 14146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c"
I20260927 15:07:50.571182 14146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 2 FOLLOWER]: Advancing to term 3
I20260927 15:07:50.571750 14675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.571527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.0.0.1:45004)
0927 15:07:50.571654 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.571699 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 15:07:50.571856 13939 leader_election.cc:334] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer bc7b3a82e94e47b28a0f16ceb4f31721 (127.7.17.254:33177): Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
I20260927 15:07:50.572253 14146 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c1aa5b16f0847f2ac9bb445628f78f7 in term 3.
I20260927 15:07:50.572518 13941 leader_election.cc:302] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c1aa5b16f0847f2ac9bb445628f78f7, d7a6e46bfaba4e33802c1991a2cc8c0c; no voters: bc7b3a82e94e47b28a0f16ceb4f31721
I20260927 15:07:50.572688 14674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 15:07:50.572875 14674 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [term 3 LEADER]: Becoming Leader. State: Replica: 3c1aa5b16f0847f2ac9bb445628f78f7, State: Running, Role: LEADER
I20260927 15:07:50.573140 14674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: false } }
I20260927 15:07:50.574157 14677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c1aa5b16f0847f2ac9bb445628f78f7. Latest consensus state: current_term: 3 leader_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: false } } }
I20260927 15:07:50.574254 14677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:50.574509 14679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:50.576591 14679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:50.577026 14679 catalog_manager.cc:1259] Loaded cluster ID: 79c4dfdb3ecf46408724e71fcbeb7be8
I20260927 15:07:50.577080 14679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:50.577608 14679 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:50.577958 14679 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7: Loaded TSK: 0
I20260927 15:07:50.578614 14679 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:07:50.611754 13946 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:07:50.614940 13946 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:50.618450 14675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.618159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.0.0.1:45026)
0927 15:07:50.618303 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.618369 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 15:07:50.618515 14146 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [term 3 FOLLOWER]: Refusing update from remote peer 3c1aa5b16f0847f2ac9bb445628f78f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 15:07:50.619014 14674 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 15:07:50.619392 14675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.619159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.0.0.1:45030)
0927 15:07:50.619268 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.619323 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 15:07:50.619711 13939 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 -> Peer bc7b3a82e94e47b28a0f16ceb4f31721 (127.7.17.254:33177): Couldn't send request to peer bc7b3a82e94e47b28a0f16ceb4f31721. Status: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 15:07:50.620576 14681 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c1aa5b16f0847f2ac9bb445628f78f7. Latest consensus state: current_term: 3 leader_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: false } } }
I20260927 15:07:50.620710 14681 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:07:50.621794 14674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: false } } }
I20260927 15:07:50.621920 14674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:50.622586 14681 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bc7b3a82e94e47b28a0f16ceb4f31721" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33177 } } peers { permanent_uuid: "3c1aa5b16f0847f2ac9bb445628f78f7" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 34217 } attrs { promote: false } } peers { permanent_uuid: "d7a6e46bfaba4e33802c1991a2cc8c0c" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 40175 } attrs { promote: false } } }
I20260927 15:07:50.622706 14681 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7a6e46bfaba4e33802c1991a2cc8c0c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:07:50.632371 14688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.627743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:45549 (local address 127.0.0.1:39676)
0927 15:07:50.628228 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.628243 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:50.628348 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:50.628656 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:50.628659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:50.628675 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:50.628789 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:50.628795 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.629779 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.629782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:50.631005 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:50.631015 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.631913 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.631917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:50.632046 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:50.632071 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:07:50.632141 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 15:07:50.632215 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":28,"spinlock_wait_cycles":8064}
I20260927 15:07:50.632364 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.627870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39676 (local address 127.7.17.193:45549)
0927 15:07:50.628465 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:50.628470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:50.628482 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:07:50.628527 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:50.628529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:50.628565 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:50.628639 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:50.628920 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.629666 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.631132 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.631690 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.631788 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:50.631859 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:50.632174 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 15:07:50.632250 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":82,"threads_started":1}
I20260927 15:07:50.632748 14675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.627967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:46871 (local address 127.0.0.1:35448)
0927 15:07:50.628229 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.628243 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:50.628335 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:50.629581 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:50.629584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:50.629599 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:50.629689 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:50.629693 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.630482 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.630485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:50.631384 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:50.631393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.632381 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.632384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:50.632483 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:50.632505 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:07:50.632556 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 15:07:50.632620 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":108,"threads_started":1}
I20260927 15:07:50.633381 14689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.628215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35448 (local address 127.7.17.194:46871)
0927 15:07:50.629355 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:50.629359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:50.629372 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:07:50.629416 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:50.629419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:50.629461 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:50.629544 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:50.629807 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.630385 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.631532 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.632244 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.633063 (+ 819us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:50.633129 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:50.633166 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 15:07:50.633239 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1063,"thread_start_us":70,"threads_started":1}
I20260927 15:07:50.633435 14249 tablet_service.cc:1467] Processing CreateTablet for tablet 6ac9aab0c1e0480992ccb98d980d916d (DEFAULT_TABLE table=test-table [id=0d7517c47f0b473e83334a74be12b209]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:50.633710 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.628218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:40949 (local address 127.0.0.1:33206)
0927 15:07:50.628500 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.628511 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:50.628592 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:50.629778 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:50.629780 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:50.629798 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:50.629932 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:50.629939 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.631115 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.631119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:50.632356 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:50.632365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.633289 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.633293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:50.633386 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:50.633409 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:07:50.633492 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 15:07:50.633562 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":23,"thread_start_us":79,"threads_started":1}
I20260927 15:07:50.633723 14691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.628528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33206 (local address 127.7.17.195:40949)
0927 15:07:50.628938 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:50.628942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:50.628955 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:07:50.629014 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:50.629017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:50.629068 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:50.629148 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:50.630115 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.630951 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.632569 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.633158 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.633259 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:50.633318 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:50.633512 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 15:07:50.633592 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":91,"threads_started":1}
I20260927 15:07:50.633767 14249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ac9aab0c1e0480992ccb98d980d916d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:50.634079 14386 tablet_service.cc:1467] Processing CreateTablet for tablet 6ac9aab0c1e0480992ccb98d980d916d (DEFAULT_TABLE table=test-table [id=0d7517c47f0b473e83334a74be12b209]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:50.634303 14386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ac9aab0c1e0480992ccb98d980d916d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:50.634281 14519 tablet_service.cc:1467] Processing CreateTablet for tablet 6ac9aab0c1e0480992ccb98d980d916d (DEFAULT_TABLE table=test-table [id=0d7517c47f0b473e83334a74be12b209]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:50.634498 14519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ac9aab0c1e0480992ccb98d980d916d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:50.636179 14692 tablet_bootstrap.cc:492] T 6ac9aab0c1e0480992ccb98d980d916d P 3a3d3d46c72546e48fdcdcc8b8996d6d: Bootstrap starting.
I20260927 15:07:50.636816 14694 tablet_bootstrap.cc:492] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec: Bootstrap starting.
I20260927 15:07:50.637365 14692 tablet_bootstrap.cc:654] T 6ac9aab0c1e0480992ccb98d980d916d P 3a3d3d46c72546e48fdcdcc8b8996d6d: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:50.637871 14694 tablet_bootstrap.cc:654] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:50.638049 14693 tablet_bootstrap.cc:492] T 6ac9aab0c1e0480992ccb98d980d916d P 627b253d79314dd39ba841bfedb233e6: Bootstrap starting.
I20260927 15:07:50.638556 14692 tablet_bootstrap.cc:492] T 6ac9aab0c1e0480992ccb98d980d916d P 3a3d3d46c72546e48fdcdcc8b8996d6d: No bootstrap required, opened a new log
I20260927 15:07:50.638643 14692 ts_tablet_manager.cc:1397] T 6ac9aab0c1e0480992ccb98d980d916d P 3a3d3d46c72546e48fdcdcc8b8996d6d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 15:07:50.638960 14694 tablet_bootstrap.cc:492] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec: No bootstrap required, opened a new log
I20260927 15:07:50.639038 14694 ts_tablet_manager.cc:1397] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:07:50.639070 14692 raft_consensus.cc:348] T 6ac9aab0c1e0480992ccb98d980d916d P 3a3d3d46c72546e48fdcdcc8b8996d6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "627b253d79314dd39ba841bfedb233e6" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 45549 } } peers { permanent_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 46871 } } peers { permanent_uuid: "be770c2746a2448e9afa82fdf76f08ec" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 40949 } }
I20260927 15:07:50.639206 14692 raft_consensus.cc:374] T 6ac9aab0c1e0480992ccb98d980d916d P 3a3d3d46c72546e48fdcdcc8b8996d6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:50.639240 14692 raft_consensus.cc:729] T 6ac9aab0c1e0480992ccb98d980d916d P 3a3d3d46c72546e48fdcdcc8b8996d6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3d3d46c72546e48fdcdcc8b8996d6d, State: Initialized, Role: FOLLOWER
I20260927 15:07:50.639376 14692 consensus_queue.cc:260] T 6ac9aab0c1e0480992ccb98d980d916d P 3a3d3d46c72546e48fdcdcc8b8996d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "627b253d79314dd39ba841bfedb233e6" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 45549 } } peers { permanent_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 46871 } } peers { permanent_uuid: "be770c2746a2448e9afa82fdf76f08ec" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 40949 } }
I20260927 15:07:50.639427 14694 raft_consensus.cc:348] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "627b253d79314dd39ba841bfedb233e6" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 45549 } } peers { permanent_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 46871 } } peers { permanent_uuid: "be770c2746a2448e9afa82fdf76f08ec" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 40949 } }
I20260927 15:07:50.639550 14694 raft_consensus.cc:374] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:50.639602 14694 raft_consensus.cc:729] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be770c2746a2448e9afa82fdf76f08ec, State: Initialized, Role: FOLLOWER
I20260927 15:07:50.639864 14692 ts_tablet_manager.cc:1428] T 6ac9aab0c1e0480992ccb98d980d916d P 3a3d3d46c72546e48fdcdcc8b8996d6d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:50.640264 14672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.640034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.194:58567)
0927 15:07:50.640129 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.640189 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 15:07:50.640916 14693 tablet_bootstrap.cc:654] T 6ac9aab0c1e0480992ccb98d980d916d P 627b253d79314dd39ba841bfedb233e6: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:50.640997 14672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.640712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.194:34237)
0927 15:07:50.640883 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.640933 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 15:07:50.642186 14454 heartbeater.cc:503] Master 127.7.17.253:34217 was elected leader, sending a full tablet report...
I20260927 15:07:50.642232 14693 tablet_bootstrap.cc:492] T 6ac9aab0c1e0480992ccb98d980d916d P 627b253d79314dd39ba841bfedb233e6: No bootstrap required, opened a new log
I20260927 15:07:50.642333 14693 ts_tablet_manager.cc:1397] T 6ac9aab0c1e0480992ccb98d980d916d P 627b253d79314dd39ba841bfedb233e6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260927 15:07:50.642758 14694 consensus_queue.cc:260] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "627b253d79314dd39ba841bfedb233e6" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 45549 } } peers { permanent_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 46871 } } peers { permanent_uuid: "be770c2746a2448e9afa82fdf76f08ec" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 40949 } }
I20260927 15:07:50.642772 14693 raft_consensus.cc:348] T 6ac9aab0c1e0480992ccb98d980d916d P 627b253d79314dd39ba841bfedb233e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "627b253d79314dd39ba841bfedb233e6" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 45549 } } peers { permanent_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 46871 } } peers { permanent_uuid: "be770c2746a2448e9afa82fdf76f08ec" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 40949 } }
I20260927 15:07:50.642886 14693 raft_consensus.cc:374] T 6ac9aab0c1e0480992ccb98d980d916d P 627b253d79314dd39ba841bfedb233e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:50.642927 14693 raft_consensus.cc:729] T 6ac9aab0c1e0480992ccb98d980d916d P 627b253d79314dd39ba841bfedb233e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 627b253d79314dd39ba841bfedb233e6, State: Initialized, Role: FOLLOWER
I20260927 15:07:50.643086 14693 consensus_queue.cc:260] T 6ac9aab0c1e0480992ccb98d980d916d P 627b253d79314dd39ba841bfedb233e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "627b253d79314dd39ba841bfedb233e6" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 45549 } } peers { permanent_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 46871 } } peers { permanent_uuid: "be770c2746a2448e9afa82fdf76f08ec" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 40949 } }
I20260927 15:07:50.643285 14694 ts_tablet_manager.cc:1428] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260927 15:07:50.643642 14693 ts_tablet_manager.cc:1428] T 6ac9aab0c1e0480992ccb98d980d916d P 627b253d79314dd39ba841bfedb233e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260927 15:07:50.643989 14671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.643729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.195:32897)
0927 15:07:50.643882 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.643929 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 15:07:50.644616 14673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.644351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.193:54053)
0927 15:07:50.644475 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.644532 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 15:07:50.645833 14671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.645635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.195:40829)
0927 15:07:50.645718 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.645769 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 15:07:50.645969 14590 heartbeater.cc:503] Master 127.7.17.253:34217 was elected leader, sending a full tablet report...
I20260927 15:07:50.646533 14673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.646211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.193:59771)
0927 15:07:50.646418 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.646468 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260927 15:07:50.646904 14317 heartbeater.cc:503] Master 127.7.17.253:34217 was elected leader, sending a full tablet report...
W20260927 15:07:50.756978 14591 tablet.cc:2369] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:07:50.807099 14455 tablet.cc:2369] T 6ac9aab0c1e0480992ccb98d980d916d P 3a3d3d46c72546e48fdcdcc8b8996d6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:07:50.877583 14699 raft_consensus.cc:482] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:50.877764 14699 raft_consensus.cc:504] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "627b253d79314dd39ba841bfedb233e6" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 45549 } } peers { permanent_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 46871 } } peers { permanent_uuid: "be770c2746a2448e9afa82fdf76f08ec" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 40949 } }
I20260927 15:07:50.878911 14699 leader_election.cc:288] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 627b253d79314dd39ba841bfedb233e6 (127.7.17.193:45549), 3a3d3d46c72546e48fdcdcc8b8996d6d (127.7.17.194:46871)
I20260927 15:07:50.882992 14671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.879156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:45549 (local address 127.7.17.195:43313)
0927 15:07:50.879304 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.879319 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:50.879439 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:50.879719 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:50.879722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:50.879747 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:50.879841 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:50.879848 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.880716 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.880719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:50.881709 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:50.881719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.882535 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.882539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:50.882662 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:50.882690 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 15:07:50.882760 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 15:07:50.882850 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":1}
I20260927 15:07:50.883039 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.879330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:43313 (local address 127.7.17.193:45549)
0927 15:07:50.879464 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:50.879469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:50.879484 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:07:50.879538 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:50.879541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:50.879584 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:50.879654 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:50.879970 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.880591 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.881847 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.882400 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.882497 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:50.882539 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:50.882816 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 15:07:50.882904 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:07:50.883183 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.879411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:46871 (local address 127.7.17.195:32811)
0927 15:07:50.879761 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.879773 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:50.879853 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:50.880083 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:50.880085 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:50.880102 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:50.880198 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:50.880204 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.881122 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.881125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:50.882008 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:50.882018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.882726 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.882729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:50.882834 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:50.882856 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:07:50.882936 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 15:07:50.883004 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":22,"thread_start_us":134,"threads_started":1}
I20260927 15:07:50.883213 14689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.879575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:32811 (local address 127.7.17.194:46871)
0927 15:07:50.879687 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:50.879691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:50.879850 (+ 159us) server_negotiation.cc:407] Connection header received
0927 15:07:50.879927 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:50.879930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:50.879985 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:50.880049 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:50.880344 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.880977 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.882150 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.882597 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.882715 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:50.882765 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:50.882960 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 15:07:50.883040 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:07:50.883836 14406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac9aab0c1e0480992ccb98d980d916d" candidate_uuid: "be770c2746a2448e9afa82fdf76f08ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" is_pre_election: true
I20260927 15:07:50.883886 14269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac9aab0c1e0480992ccb98d980d916d" candidate_uuid: "be770c2746a2448e9afa82fdf76f08ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "627b253d79314dd39ba841bfedb233e6" is_pre_election: true
I20260927 15:07:50.884002 14406 raft_consensus.cc:2454] T 6ac9aab0c1e0480992ccb98d980d916d P 3a3d3d46c72546e48fdcdcc8b8996d6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be770c2746a2448e9afa82fdf76f08ec in term 0.
I20260927 15:07:50.884123 14269 raft_consensus.cc:2454] T 6ac9aab0c1e0480992ccb98d980d916d P 627b253d79314dd39ba841bfedb233e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be770c2746a2448e9afa82fdf76f08ec in term 0.
I20260927 15:07:50.884454 14476 leader_election.cc:302] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3d3d46c72546e48fdcdcc8b8996d6d, be770c2746a2448e9afa82fdf76f08ec; no voters:
I20260927 15:07:50.884718 14699 raft_consensus.cc:2781] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:50.884778 14699 raft_consensus.cc:482] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:50.884819 14699 raft_consensus.cc:3037] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:50.885838 14699 raft_consensus.cc:504] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "627b253d79314dd39ba841bfedb233e6" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 45549 } } peers { permanent_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 46871 } } peers { permanent_uuid: "be770c2746a2448e9afa82fdf76f08ec" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 40949 } }
I20260927 15:07:50.886201 14699 leader_election.cc:288] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec [CANDIDATE]: Term 1 election: Requested vote from peers 627b253d79314dd39ba841bfedb233e6 (127.7.17.193:45549), 3a3d3d46c72546e48fdcdcc8b8996d6d (127.7.17.194:46871)
I20260927 15:07:50.886638 14406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac9aab0c1e0480992ccb98d980d916d" candidate_uuid: "be770c2746a2448e9afa82fdf76f08ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d"
I20260927 15:07:50.886631 14269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac9aab0c1e0480992ccb98d980d916d" candidate_uuid: "be770c2746a2448e9afa82fdf76f08ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "627b253d79314dd39ba841bfedb233e6"
I20260927 15:07:50.886755 14406 raft_consensus.cc:3037] T 6ac9aab0c1e0480992ccb98d980d916d P 3a3d3d46c72546e48fdcdcc8b8996d6d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:50.886762 14269 raft_consensus.cc:3037] T 6ac9aab0c1e0480992ccb98d980d916d P 627b253d79314dd39ba841bfedb233e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:50.887640 14406 raft_consensus.cc:2454] T 6ac9aab0c1e0480992ccb98d980d916d P 3a3d3d46c72546e48fdcdcc8b8996d6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be770c2746a2448e9afa82fdf76f08ec in term 1.
I20260927 15:07:50.887800 14269 raft_consensus.cc:2454] T 6ac9aab0c1e0480992ccb98d980d916d P 627b253d79314dd39ba841bfedb233e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be770c2746a2448e9afa82fdf76f08ec in term 1.
I20260927 15:07:50.887953 14476 leader_election.cc:302] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3d3d46c72546e48fdcdcc8b8996d6d, be770c2746a2448e9afa82fdf76f08ec; no voters:
I20260927 15:07:50.888118 14699 raft_consensus.cc:2781] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:50.888343 14699 raft_consensus.cc:686] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec [term 1 LEADER]: Becoming Leader. State: Replica: be770c2746a2448e9afa82fdf76f08ec, State: Running, Role: LEADER
I20260927 15:07:50.888618 14699 consensus_queue.cc:237] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "627b253d79314dd39ba841bfedb233e6" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 45549 } } peers { permanent_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 46871 } } peers { permanent_uuid: "be770c2746a2448e9afa82fdf76f08ec" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 40949 } }
I20260927 15:07:50.889153 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.888901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.195:45275)
0927 15:07:50.889021 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.889082 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 15:07:50.891382 13948 catalog_manager.cc:5495] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec reported cstate change: term changed from 0 to 1, leader changed from <none> to be770c2746a2448e9afa82fdf76f08ec (127.7.17.195). New cstate: current_term: 1 leader_uuid: "be770c2746a2448e9afa82fdf76f08ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "627b253d79314dd39ba841bfedb233e6" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 45549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 46871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be770c2746a2448e9afa82fdf76f08ec" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 40949 } health_report { overall_health: HEALTHY } } }
I20260927 15:07:50.931198 14691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.927658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33214 (local address 127.7.17.195:40949)
0927 15:07:50.927796 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:50.927800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:50.927816 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:07:50.927897 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:50.927899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:50.927969 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:50.928065 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:50.928314 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.928911 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.929769 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:50.929983 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:50.930069 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:50.930336 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:50.930812 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:50.930917 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:50.930991 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 15:07:50.931053 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:07:50.934275 14406 raft_consensus.cc:1261] T 6ac9aab0c1e0480992ccb98d980d916d P 3a3d3d46c72546e48fdcdcc8b8996d6d [term 1 FOLLOWER]: Refusing update from remote peer be770c2746a2448e9afa82fdf76f08ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:50.934613 14269 raft_consensus.cc:1261] T 6ac9aab0c1e0480992ccb98d980d916d P 627b253d79314dd39ba841bfedb233e6 [term 1 FOLLOWER]: Refusing update from remote peer be770c2746a2448e9afa82fdf76f08ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:50.934863 14702 consensus_queue.cc:1035] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3d3d46c72546e48fdcdcc8b8996d6d" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 46871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:50.935166 14699 consensus_queue.cc:1035] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "627b253d79314dd39ba841bfedb233e6" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 45549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:50.936622 14672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.936316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.194:48489)
0927 15:07:50.936456 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.936528 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 15:07:50.937453 14673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.937134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.193:54657)
0927 15:07:50.937316 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.937375 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260927 15:07:50.939370 14705 mvcc.cc:205] Tried to move back new op lower bound from 7333976764141588480 to 7333976763962974208. Current Snapshot: MvccSnapshot[applied={T|T < 7333976764141588480}]
I20260927 15:07:50.940179 14672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.939862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.194:49173)
0927 15:07:50.940036 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.940105 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260927 15:07:50.942068 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.941828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.195:46535)
0927 15:07:50.941932 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.941992 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 15:07:50.943128 14707 mvcc.cc:205] Tried to move back new op lower bound from 7333976764141588480 to 7333976763962974208. Current Snapshot: MvccSnapshot[applied={T|T < 7333976764141588480}]
I20260927 15:07:50.943368 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.942910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.195:36783)
0927 15:07:50.943248 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.943300 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262}
I20260927 15:07:50.943518 14673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.943178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33177 (local address 127.7.17.193:59915)
0927 15:07:50.943411 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.943458 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
I20260927 15:07:50.948587 13946 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:50.948800 13946 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:50.953174 13946 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c1aa5b16f0847f2ac9bb445628f78f7: Sending DeleteTablet for 3 replicas of tablet 6ac9aab0c1e0480992ccb98d980d916d
I20260927 15:07:50.954053 14386 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ac9aab0c1e0480992ccb98d980d916d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:50 UTC) from {username='oryx', principal='oryx/127.7.17.253@KRBTEST.COM'} at 127.0.0.1:35448
I20260927 15:07:50.954105 14519 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ac9aab0c1e0480992ccb98d980d916d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:50 UTC) from {username='oryx', principal='oryx/127.7.17.253@KRBTEST.COM'} at 127.0.0.1:33206
I20260927 15:07:50.954598 14249 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ac9aab0c1e0480992ccb98d980d916d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:50 UTC) from {username='oryx', principal='oryx/127.7.17.253@KRBTEST.COM'} at 127.0.0.1:39676
I20260927 15:07:50.954926 14717 tablet_replica.cc:330] T 6ac9aab0c1e0480992ccb98d980d916d P 627b253d79314dd39ba841bfedb233e6: stopping tablet replica
I20260927 15:07:50.955057 14717 raft_consensus.cc:2229] T 6ac9aab0c1e0480992ccb98d980d916d P 627b253d79314dd39ba841bfedb233e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:50.955197 14717 raft_consensus.cc:2258] T 6ac9aab0c1e0480992ccb98d980d916d P 627b253d79314dd39ba841bfedb233e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:50.955511 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 14098
I20260927 15:07:50.961063 14716 tablet_replica.cc:330] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec: stopping tablet replica
I20260927 15:07:50.961297 14716 raft_consensus.cc:2229] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:07:50.961731 14716 raft_consensus.cc:2258] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:50.962492 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 14323
I20260927 15:07:50.963208 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.962626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:45549 (local address 127.0.0.1:39688)
0927 15:07:50.963061 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.963113 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.193:45549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357}
W20260927 15:07:50.963371 13939 catalog_manager.cc:4584] TS 627b253d79314dd39ba841bfedb233e6 (127.7.17.193:45549): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6ac9aab0c1e0480992ccb98d980d916d: Network error: Client connection negotiation failed: client connection to 127.7.17.193:45549: connect: Connection refused (error 111)
I20260927 15:07:50.964310 14716 ts_tablet_manager.cc:1905] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:50.966531 14716 ts_tablet_manager.cc:1918] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:50.966614 14716 log.cc:1196] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/6ac9aab0c1e0480992ccb98d980d916d
I20260927 15:07:50.966876 14716 ts_tablet_manager.cc:1939] T 6ac9aab0c1e0480992ccb98d980d916d P be770c2746a2448e9afa82fdf76f08ec: Deleting consensus metadata
I20260927 15:07:50.967720 13940 catalog_manager.cc:4841] TS be770c2746a2448e9afa82fdf76f08ec (127.7.17.195:40949): tablet 6ac9aab0c1e0480992ccb98d980d916d (table test-table [id=0d7517c47f0b473e83334a74be12b209]) successfully deleted
I20260927 15:07:50.970793 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 14459
I20260927 15:07:50.971786 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:50.971005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:46871 (local address 127.0.0.1:35456)
0927 15:07:50.971624 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:50.971694 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.194:46871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528}
I20260927 15:07:50.976214 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 13816
I20260927 15:07:50.982390 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 13926
2026-09-27T15:07:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7039 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:07:51 dist-test-slave-hhpq krb5kdc[14720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:07:51 dist-test-slave-hhpq krb5kdc[14720](info): set up 2 sockets
Sep 27 15:07:51 dist-test-slave-hhpq krb5kdc[14720](info): commencing operation
krb5kdc: starting...
W20260927 15:07:51.941277 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:07:52 dist-test-slave-hhpq krb5kdc[14720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:07:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.254.keytab.
Entry for principal oryx/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.254.keytab.
I20260927 15:07:52.064625 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:40831
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.254.keytab
--principal=oryx/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:40555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:40831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:52.175705 14733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:52.176025 14733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:52.176069 14733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:52.180224 14733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:52.180305 14733 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 15:07:52.180321 14733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:52.180337 14733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:52.180353 14733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:52.180367 14733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:52.185868 14733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:40555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:40831
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.254.keytab
--principal=oryx/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:40831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:52.186930 14733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:52.187906 14733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:52.196206 14738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:52.196208 14739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:52.196307 14741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:52.196494 14733 server_base.cc:1034] running on GCE node
I20260927 15:07:52.197198 14733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:52.198424 14733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:52.199590 14733 hybrid_clock.cc:648] HybridClock initialized: now 1790521672199577 us; error 28 us; skew 500 ppm
Sep 27 15:07:52 dist-test-slave-hhpq krb5kdc[14720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521672, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:52.202646 14733 init.cc:377] Logged in from keytab as oryx/127.7.17.254@KRBTEST.COM (short username oryx)
I20260927 15:07:52.203608 14733 webserver.cc:460] Webserver started at http://127.7.17.254:40789/ using document root <none> and password file <none>
I20260927 15:07:52.204186 14733 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:52.204257 14733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:52.204457 14733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:52.206664 14733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "dd7c17b011a8423a849d956de0960bc9"
format_stamp: "Formatted at 2026-09-27 15:07:52 on dist-test-slave-hhpq"
server_key: "4f608c9867046506236dd10f58cb2e1e"
server_key_iv: "a8ed0d9accd4cb2619ef06bccaba48be"
server_key_version: "encryptionkey@0"
I20260927 15:07:52.207278 14733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "dd7c17b011a8423a849d956de0960bc9"
format_stamp: "Formatted at 2026-09-27 15:07:52 on dist-test-slave-hhpq"
server_key: "4f608c9867046506236dd10f58cb2e1e"
server_key_iv: "a8ed0d9accd4cb2619ef06bccaba48be"
server_key_version: "encryptionkey@0"
I20260927 15:07:52.211234 14733 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:07:52.213719 14748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:52.215026 14733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:07:52.215152 14733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "dd7c17b011a8423a849d956de0960bc9"
format_stamp: "Formatted at 2026-09-27 15:07:52 on dist-test-slave-hhpq"
server_key: "4f608c9867046506236dd10f58cb2e1e"
server_key_iv: "a8ed0d9accd4cb2619ef06bccaba48be"
server_key_version: "encryptionkey@0"
I20260927 15:07:52.215265 14733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:52.227599 14733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:52.230645 14733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:52.230834 14733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:52.238394 14733 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:40831
I20260927 15:07:52.238425 14810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:40831 every 8 connection(s)
I20260927 15:07:52.239344 14733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:52.240895 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 14733
I20260927 15:07:52.241019 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:07:52.241349 7239 external_mini_cluster.cc:1428] Setting key 654aa6b24d2e4f2c0947fb2572e10434
I20260927 15:07:52.242549 14811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:52.248715 14811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9: Bootstrap starting.
Sep 27 15:07:52 dist-test-slave-hhpq krb5kdc[14720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM
I20260927 15:07:52.251279 14811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:52.252334 14811 log.cc:824] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:52.254248 14811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9: No bootstrap required, opened a new log
I20260927 15:07:52.255214 14814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:52.242842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46642 (local address 127.7.17.254:40831)
0927 15:07:52.243272 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:52.243287 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:52.243322 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:07:52.243954 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:52.243971 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:52.244337 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:52.244655 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:52.245469 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.246476 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.247307 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.247784 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.247924 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:52.250594 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:52.250599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:52.250614 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:52.250655 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:52.252450 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:52.252914 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:52.252920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:52.252926 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:52.253041 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:52.253307 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:52.253310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:52.253312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:52.253715 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:52.253895 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:52.254183 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 15:07:52.254474 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":137,"threads_started":1}
I20260927 15:07:52.257416 14811 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7c17b011a8423a849d956de0960bc9" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 40831 } }
I20260927 15:07:52.257666 14811 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:52.257727 14811 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7c17b011a8423a849d956de0960bc9, State: Initialized, Role: FOLLOWER
I20260927 15:07:52.258128 14811 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7c17b011a8423a849d956de0960bc9" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 40831 } }
I20260927 15:07:52.258244 14811 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:52.258301 14811 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:52.258399 14811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:52.259269 14811 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7c17b011a8423a849d956de0960bc9" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 40831 } }
I20260927 15:07:52.259581 14811 leader_election.cc:302] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd7c17b011a8423a849d956de0960bc9; no voters:
I20260927 15:07:52.259871 14811 leader_election.cc:288] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:52.259964 14816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:52.260125 14816 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9 [term 1 LEADER]: Becoming Leader. State: Replica: dd7c17b011a8423a849d956de0960bc9, State: Running, Role: LEADER
I20260927 15:07:52.260414 14816 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7c17b011a8423a849d956de0960bc9" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 40831 } }
I20260927 15:07:52.261198 14811 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:52.262228 14818 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd7c17b011a8423a849d956de0960bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7c17b011a8423a849d956de0960bc9" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 40831 } } }
I20260927 15:07:52.262945 14818 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:52.262377 14817 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd7c17b011a8423a849d956de0960bc9. Latest consensus state: current_term: 1 leader_uuid: "dd7c17b011a8423a849d956de0960bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7c17b011a8423a849d956de0960bc9" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 40831 } } }
I20260927 15:07:52.263399 14817 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9 [sys.catalog]: This master's current role is: LEADER
W20260927 15:07:52.266527 14831 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:07:52.266605 14831 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:07:52.266680 14830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:52.267372 14830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:52.273260 14830 catalog_manager.cc:1347] Generated new cluster ID: 3fdeff38ee424ee19afd862a78f54597
I20260927 15:07:52.273342 14830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:52.300115 14830 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:52.301548 14830 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:52.316709 14830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9: Generated new TSK 0
I20260927 15:07:52.317553 14830 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.193.keytab.
Entry for principal oryx/127.7.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.193.keytab.
I20260927 15:07:52.357678 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:40831
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.193.keytab
--principal=oryx/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:40555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:52.473073 14837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:52.473358 14837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:52.473403 14837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:52.477600 14837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:52.477681 14837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:52.477722 14837 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:07:52.477759 14837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:07:52.483361 14837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:40555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.193.keytab
--principal=oryx/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.17.254:40831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:52.484469 14837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:52.485567 14837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:52.495115 14842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:52.495173 14837 server_base.cc:1034] running on GCE node
W20260927 15:07:52.495146 14845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:52.495139 14843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:52.495779 14837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:52.496390 14837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:52.497565 14837 hybrid_clock.cc:648] HybridClock initialized: now 1790521672497541 us; error 39 us; skew 500 ppm
Sep 27 15:07:52 dist-test-slave-hhpq krb5kdc[14720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521672, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:52.500449 14837 init.cc:377] Logged in from keytab as oryx/127.7.17.193@KRBTEST.COM (short username oryx)
I20260927 15:07:52.501432 14837 webserver.cc:460] Webserver started at http://127.7.17.193:38337/ using document root <none> and password file <none>
I20260927 15:07:52.501988 14837 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:52.502054 14837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:52.502252 14837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:52.504420 14837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "36a116f98c2d4cb8b337e3a71a180e8e"
format_stamp: "Formatted at 2026-09-27 15:07:52 on dist-test-slave-hhpq"
server_key: "4b2ec8d5802026287e1ebe58d6b2d797"
server_key_iv: "c58ac6dc580dd1082fc35aa09615f08d"
server_key_version: "encryptionkey@0"
I20260927 15:07:52.505064 14837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "36a116f98c2d4cb8b337e3a71a180e8e"
format_stamp: "Formatted at 2026-09-27 15:07:52 on dist-test-slave-hhpq"
server_key: "4b2ec8d5802026287e1ebe58d6b2d797"
server_key_iv: "c58ac6dc580dd1082fc35aa09615f08d"
server_key_version: "encryptionkey@0"
I20260927 15:07:52.509073 14837 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:07:52.511888 14852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:52.513304 14837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:52.513427 14837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "36a116f98c2d4cb8b337e3a71a180e8e"
format_stamp: "Formatted at 2026-09-27 15:07:52 on dist-test-slave-hhpq"
server_key: "4b2ec8d5802026287e1ebe58d6b2d797"
server_key_iv: "c58ac6dc580dd1082fc35aa09615f08d"
server_key_version: "encryptionkey@0"
I20260927 15:07:52.513561 14837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:52.525125 14837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:52.528359 14837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:52.528537 14837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:52.530088 14837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:52.530143 14837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:52.530195 14837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:52.530224 14837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:07:52 dist-test-slave-hhpq krb5kdc[14720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521672, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM
I20260927 15:07:52.539984 14837 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:41869
I20260927 15:07:52.540352 14967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:41869 every 8 connection(s)
I20260927 15:07:52.541009 14837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:52.543051 14814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:52.530833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:52019 (local address 127.7.17.254:40831)
0927 15:07:52.530968 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:52.530973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:52.531875 (+ 902us) server_negotiation.cc:407] Connection header received
0927 15:07:52.532980 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:52.532984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:52.533045 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:52.533106 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:52.534630 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.535255 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.536494 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.536743 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.536873 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:52.539481 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:52.539484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:52.539486 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:52.539512 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:52.541002 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:52.541487 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:52.541491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:52.541493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:52.541586 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:52.541960 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:52.541963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:52.541966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:52.542181 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:52.542269 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:52.542758 (+ 489us) server_negotiation.cc:299] Negotiation successful
0927 15:07:52.542890 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:07:52.543813 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 14837
I20260927 15:07:52.543823 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:52.531256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:40831 (local address 127.7.17.193:52019)
0927 15:07:52.531702 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:52.531754 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:52.532692 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:52.533515 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:52.533525 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:52.533893 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:52.534456 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:52.534476 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.535382 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.535384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:52.536379 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:52.536389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.536931 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.536935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:52.537103 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:52.537720 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:52.537737 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:52.539308 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:52.541115 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:52.541119 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:52.541131 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:52.541376 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:52.541700 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:52.541703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:52.541705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:52.541864 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:52.542281 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:52.542288 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:52.542519 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 15:07:52.542701 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 15:07:52.543021 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":113,"threads_started":1}
I20260927 15:07:52.543931 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:07:52.544184 7239 external_mini_cluster.cc:1428] Setting key 6104e2ffaa0a0c0254349472fc98fdbd
I20260927 15:07:52.545215 14968 heartbeater.cc:347] Connected to a master server at 127.7.17.254:40831
I20260927 15:07:52.545368 14968 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:52.545948 14968 heartbeater.cc:511] Master 127.7.17.254:40831 requested a full tablet report, sending...
I20260927 15:07:52.547609 14764 ts_manager.cc:194] Registered new tserver with Master: 36a116f98c2d4cb8b337e3a71a180e8e (127.7.17.193:41869)
I20260927 15:07:52.548751 14764 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:52019
I20260927 15:07:52.554502 14814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:52.547190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:56823 (local address 127.7.17.254:40831)
0927 15:07:52.547293 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:52.547297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:52.547586 (+ 289us) server_negotiation.cc:407] Connection header received
0927 15:07:52.547671 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:52.547673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:52.547714 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:52.547797 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:52.548272 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.548774 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.549723 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.550041 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.550141 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:52.551358 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:52.551362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:52.551364 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:52.551390 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:52.552940 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:52.553359 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:52.553362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:52.553364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:52.553449 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:52.553701 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:52.553704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:52.553706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:52.553880 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:52.553979 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:52.554241 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 15:07:52.554349 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:07:52.554536 14975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:52.547125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:40831 (local address 127.7.17.193:56823)
0927 15:07:52.547484 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:52.547498 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:52.547594 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:52.547897 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:52.547899 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:52.548061 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:52.548154 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:52.548159 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.548899 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.548902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:52.549604 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:52.549615 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.550161 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.550164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:52.550292 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:52.550803 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:52.550817 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:52.551225 (+ 408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:52.553060 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:52.553064 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:52.553066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:52.553269 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:52.553534 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:52.553537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:52.553539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:52.553620 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:52.553970 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:52.553973 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:52.554088 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 15:07:52.554176 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 15:07:52.554323 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for oryx/127.7.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.194.keytab.
Entry for principal oryx/127.7.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.194.keytab.
I20260927 15:07:52.575881 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:40831
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.194.keytab
--principal=oryx/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:40555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:52.686487 14977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:52.686782 14977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:52.686827 14977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:52.690989 14977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:52.691066 14977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:52.691123 14977 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:07:52.691176 14977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:07:52.696897 14977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:40555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.194.keytab
--principal=oryx/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.17.254:40831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:52.698033 14977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:52.699041 14977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:52.708907 14982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:52.708968 14977 server_base.cc:1034] running on GCE node
W20260927 15:07:52.709189 14983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:52.709318 14985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:52.709767 14977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:52.710457 14977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:52.711622 14977 hybrid_clock.cc:648] HybridClock initialized: now 1790521672711599 us; error 37 us; skew 500 ppm
Sep 27 15:07:52 dist-test-slave-hhpq krb5kdc[14720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521672, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:52.714401 14977 init.cc:377] Logged in from keytab as oryx/127.7.17.194@KRBTEST.COM (short username oryx)
I20260927 15:07:52.715274 14977 webserver.cc:460] Webserver started at http://127.7.17.194:37621/ using document root <none> and password file <none>
I20260927 15:07:52.715813 14977 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:52.715875 14977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:52.716073 14977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:52.718400 14977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "6b1c2b30413046bbb45b6d321ddab1a2"
format_stamp: "Formatted at 2026-09-27 15:07:52 on dist-test-slave-hhpq"
server_key: "995daa755dbdc40158598f9a6062bcc9"
server_key_iv: "a9bd065465353f216f3ec4607409d0e2"
server_key_version: "encryptionkey@0"
I20260927 15:07:52.719020 14977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "6b1c2b30413046bbb45b6d321ddab1a2"
format_stamp: "Formatted at 2026-09-27 15:07:52 on dist-test-slave-hhpq"
server_key: "995daa755dbdc40158598f9a6062bcc9"
server_key_iv: "a9bd065465353f216f3ec4607409d0e2"
server_key_version: "encryptionkey@0"
I20260927 15:07:52.723042 14977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:07:52.725889 14992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:52.727309 14977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:52.727450 14977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "6b1c2b30413046bbb45b6d321ddab1a2"
format_stamp: "Formatted at 2026-09-27 15:07:52 on dist-test-slave-hhpq"
server_key: "995daa755dbdc40158598f9a6062bcc9"
server_key_iv: "a9bd065465353f216f3ec4607409d0e2"
server_key_version: "encryptionkey@0"
I20260927 15:07:52.727568 14977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:52.742637 14977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:52.745940 14977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:52.746126 14977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:52.747601 14977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:52.747655 14977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:52.747716 14977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:52.747745 14977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:07:52 dist-test-slave-hhpq krb5kdc[14720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521672, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM
I20260927 15:07:52.758323 14977 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:39663
I20260927 15:07:52.758405 15107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:39663 every 8 connection(s)
I20260927 15:07:52.759313 14977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:07:52.762434 14814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:52.749231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:43277 (local address 127.7.17.254:40831)
0927 15:07:52.749363 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:52.749368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:52.749873 (+ 505us) server_negotiation.cc:407] Connection header received
0927 15:07:52.751120 (+ 1247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:52.751124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:52.751185 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:52.751281 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:52.753033 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.753698 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.755446 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.755668 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.755756 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:52.758788 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:52.758793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:52.758795 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:52.758824 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:52.760288 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:52.760881 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:52.760885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:52.760887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:52.760970 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:52.761293 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:52.761297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:52.761299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:52.761486 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:52.761569 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:52.762137 (+ 568us) server_negotiation.cc:299] Negotiation successful
0927 15:07:52.762268 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:07:52.762549 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 14977
I20260927 15:07:52.762653 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:07:52.762897 7239 external_mini_cluster.cc:1428] Setting key b377805f7797ee2b7273a5b04a4896e3
I20260927 15:07:52.763396 15001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:52.749132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:40831 (local address 127.7.17.194:43277)
0927 15:07:52.749691 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:52.749753 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:52.750856 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:52.751510 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:52.751521 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:52.752036 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:52.752845 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:52.752870 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.753824 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.753827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:52.755307 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:52.755319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.755843 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.755847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:52.756062 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:52.756852 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:52.756877 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:52.758601 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:52.760403 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:52.760409 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:52.760425 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:52.760757 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:52.761053 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:52.761056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:52.761058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:52.761197 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:52.761590 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:52.761597 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:52.761879 (+ 282us) client_negotiation.cc:769] Sending connection context
0927 15:07:52.762056 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 15:07:52.762410 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":142,"threads_started":1}
I20260927 15:07:52.765108 15108 heartbeater.cc:347] Connected to a master server at 127.7.17.254:40831
I20260927 15:07:52.765278 15108 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:52.765889 15108 heartbeater.cc:511] Master 127.7.17.254:40831 requested a full tablet report, sending...
I20260927 15:07:52.767096 14765 ts_manager.cc:194] Registered new tserver with Master: 6b1c2b30413046bbb45b6d321ddab1a2 (127.7.17.194:39663)
I20260927 15:07:52.767844 14765 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:43277
WARNING: no policy specified for oryx/127.7.17.195@KRBTEST.COM; defaulting to no policy
I20260927 15:07:52.774523 15115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:52.767305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:40831 (local address 127.7.17.194:41401)
0927 15:07:52.767579 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:52.767594 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:52.767696 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:52.767913 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:52.767916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:52.768135 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:52.768259 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:52.768266 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.769061 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.769064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:52.769786 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:52.769798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.770242 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.770246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:52.770338 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:52.770892 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:52.770903 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:52.771246 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:52.772915 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:52.772919 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:52.772921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:52.773130 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:52.773387 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:52.773390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:52.773392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:52.773478 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:52.774039 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:52.774042 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:52.774130 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 15:07:52.774224 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 15:07:52.774344 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":102,"threads_started":1}
I20260927 15:07:52.774703 14814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:52.767353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:41401 (local address 127.7.17.254:40831)
0927 15:07:52.767477 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:52.767481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:52.767661 (+ 180us) server_negotiation.cc:407] Connection header received
0927 15:07:52.767773 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:52.767775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:52.767813 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:52.767876 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:52.768367 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.768923 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.769901 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.770117 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.770203 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:52.771388 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:52.771391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:52.771393 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:52.771415 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:52.772768 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:52.773224 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:52.773228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:52.773229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:52.773305 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:52.773592 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:52.773596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:52.773598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:52.773876 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:52.773967 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:52.774397 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 15:07:52.774533 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.195.keytab.
Entry for principal oryx/127.7.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.195.keytab.
I20260927 15:07:52.791601 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:40831
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.195.keytab
--principal=oryx/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:40555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:52.904279 15117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:52.904577 15117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:52.904640 15117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:52.909148 15117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:52.909228 15117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:52.909284 15117 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:07:52.909339 15117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:07:52.914808 15117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:40555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.195.keytab
--principal=oryx/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.17.254:40831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:52.916046 15117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:52.917275 15117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:52.927534 15125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:52.927628 15122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:52.927675 15117 server_base.cc:1034] running on GCE node
W20260927 15:07:52.927794 15123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:52.928354 15117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:52.929090 15117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:52.930274 15117 hybrid_clock.cc:648] HybridClock initialized: now 1790521672930239 us; error 44 us; skew 500 ppm
Sep 27 15:07:52 dist-test-slave-hhpq krb5kdc[14720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521672, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:52.933379 15117 init.cc:377] Logged in from keytab as oryx/127.7.17.195@KRBTEST.COM (short username oryx)
I20260927 15:07:52.934368 15117 webserver.cc:460] Webserver started at http://127.7.17.195:33899/ using document root <none> and password file <none>
I20260927 15:07:52.934947 15117 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:52.935015 15117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:52.935216 15117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:52.937479 15117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "5ef67695d9fb4e979f286cb81e737490"
format_stamp: "Formatted at 2026-09-27 15:07:52 on dist-test-slave-hhpq"
server_key: "8e7cfbcbad3a920120c5e68ae05da5f9"
server_key_iv: "5b0ae240b216767fcfa9091e5636b75b"
server_key_version: "encryptionkey@0"
I20260927 15:07:52.938082 15117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "5ef67695d9fb4e979f286cb81e737490"
format_stamp: "Formatted at 2026-09-27 15:07:52 on dist-test-slave-hhpq"
server_key: "8e7cfbcbad3a920120c5e68ae05da5f9"
server_key_iv: "5b0ae240b216767fcfa9091e5636b75b"
server_key_version: "encryptionkey@0"
I20260927 15:07:52.942101 15117 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:07:52.945062 15132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:52.946571 15117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:52.946692 15117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "5ef67695d9fb4e979f286cb81e737490"
format_stamp: "Formatted at 2026-09-27 15:07:52 on dist-test-slave-hhpq"
server_key: "8e7cfbcbad3a920120c5e68ae05da5f9"
server_key_iv: "5b0ae240b216767fcfa9091e5636b75b"
server_key_version: "encryptionkey@0"
I20260927 15:07:52.946803 15117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:52.958858 15117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:52.962005 15117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:52.962180 15117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:52.963805 15117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:52.963855 15117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:52.963907 15117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:52.963936 15117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:07:52 dist-test-slave-hhpq krb5kdc[14720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521672, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM
I20260927 15:07:52.974375 15117 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:41521
I20260927 15:07:52.974404 15247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:41521 every 8 connection(s)
I20260927 15:07:52.975528 15117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:52.977455 14814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:52.964658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:50129 (local address 127.7.17.254:40831)
0927 15:07:52.964780 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:52.964783 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:52.965985 (+ 1202us) server_negotiation.cc:407] Connection header received
0927 15:07:52.967145 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:52.967149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:52.967211 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:52.967287 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:52.968690 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.969288 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.970460 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.970688 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.970789 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:52.973704 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:52.973707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:52.973709 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:52.973736 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:52.975300 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:52.975870 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:52.975875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:52.975877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:52.975978 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:52.976372 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:52.976375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:52.976377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:52.976618 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:52.976704 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:52.977226 (+ 522us) server_negotiation.cc:299] Negotiation successful
0927 15:07:52.977335 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:07:52.978210 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 15117
I20260927 15:07:52.978276 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:52.965272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:40831 (local address 127.7.17.195:50129)
0927 15:07:52.965816 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:52.965877 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:52.966883 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:52.967486 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:52.967495 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:52.967918 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:52.968524 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:52.968539 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.969409 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.969412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:52.970338 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:52.970347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.970853 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.970857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:52.971061 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:52.971686 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:52.971706 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:52.973511 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:52.975436 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:52.975441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:52.975454 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:52.975765 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:52.976091 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:52.976094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:52.976096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:52.976265 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:52.976715 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:52.976720 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:52.976999 (+ 279us) client_negotiation.cc:769] Sending connection context
0927 15:07:52.977160 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 15:07:52.977427 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":144,"threads_started":1}
I20260927 15:07:52.978315 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:07:52.978602 7239 external_mini_cluster.cc:1428] Setting key a456d1e18710b82b0aefcca0ca778fd3
I20260927 15:07:52.979583 15248 heartbeater.cc:347] Connected to a master server at 127.7.17.254:40831
I20260927 15:07:52.979739 15248 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:52.980348 15248 heartbeater.cc:511] Master 127.7.17.254:40831 requested a full tablet report, sending...
I20260927 15:07:52.981391 14764 ts_manager.cc:194] Registered new tserver with Master: 5ef67695d9fb4e979f286cb81e737490 (127.7.17.195:41521)
I20260927 15:07:52.982137 14764 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:50129
I20260927 15:07:52.988289 14814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:52.981797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:41241 (local address 127.7.17.254:40831)
0927 15:07:52.981902 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:52.981907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:52.982081 (+ 174us) server_negotiation.cc:407] Connection header received
0927 15:07:52.982179 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:52.982182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:52.982226 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:52.982291 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:52.982712 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.983210 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.983995 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.984207 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.984289 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:52.985376 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:52.985379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:52.985382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:52.985403 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:52.986735 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:52.987178 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:52.987181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:52.987183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:52.987262 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:52.987505 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:52.987508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:52.987510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:52.987685 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:52.987767 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:52.988039 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 15:07:52.988143 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 15:07:52.988302 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:52.981735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:40831 (local address 127.7.17.195:41241)
0927 15:07:52.982010 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:52.982022 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:52.982109 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:52.982305 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:52.982308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:52.982506 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:52.982608 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:52.982613 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.983341 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.983344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:52.983896 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:52.983903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:52.984298 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:52.984301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:52.984388 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:52.984816 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:52.984862 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:52.985254 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:52.986852 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:52.986856 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:52.986858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:52.987085 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:52.987343 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:52.987346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:52.987348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:52.987428 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:52.987772 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:52.987775 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:52.987878 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 15:07:52.987964 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 15:07:52.988113 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":108,"threads_started":1}
I20260927 15:07:52.992426 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:07:52 dist-test-slave-hhpq krb5kdc[14720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521672, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:07:53 dist-test-slave-hhpq krb5kdc[14720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:53 dist-test-slave-hhpq krb5kdc[14720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 15:07:53.015547 14814 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:53.009052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46644 (local address 127.7.17.254:40831)
0927 15:07:53.009251 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.009255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.009270 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:07:53.009314 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.009316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.009357 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.009452 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:53.010210 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.010728 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.011634 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.011858 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.011947 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.015386 (+ 3439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46644: BlockingRecv error: recv got EOF from 127.0.0.1:46644 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 27 15:07:53 dist-test-slave-hhpq krb5kdc[14720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521672, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM
I20260927 15:07:53.025213 14814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.017036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46652 (local address 127.7.17.254:40831)
0927 15:07:53.017203 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.017207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.017424 (+ 217us) server_negotiation.cc:407] Connection header received
0927 15:07:53.017513 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.017516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.017563 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.017630 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:53.018254 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.018823 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.019698 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.019922 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.020001 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.022379 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:53.022382 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:53.022384 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:53.022413 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:53.023737 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:53.024145 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:53.024148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:53.024149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:53.024222 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:53.024410 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:53.024411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:53.024413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:53.024575 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:53.024691 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:53.024960 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 15:07:53.025055 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 15:07:53.028393 14764 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:07:53.031518 14764 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:53.046164 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.040656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:41869 (local address 127.0.0.1:54940)
0927 15:07:53.041075 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:53.041133 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:53.041264 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:53.042067 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:53.042073 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:53.042099 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:53.042253 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.042260 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.043450 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.043454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.044727 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.044736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.045619 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.045622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.045728 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.045769 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 15:07:53.045880 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 15:07:53.045999 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":90,"threads_started":1}
I20260927 15:07:53.046157 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.040854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54940 (local address 127.7.17.193:41869)
0927 15:07:53.041552 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.041567 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.041591 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:07:53.041663 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.041668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.041889 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.042401 (+ 512us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:53.042509 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.043349 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.044884 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.045506 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.045654 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.045813 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:53.045916 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 15:07:53.046011 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":91,"threads_started":1}
I20260927 15:07:53.047076 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.041914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.7.17.194:39663)
0927 15:07:53.042275 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.042285 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.042308 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:07:53.042369 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.042373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.042522 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.042676 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:53.042892 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.044033 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.045535 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.046458 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.046559 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.046710 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:53.046849 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 15:07:53.046936 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":88,"threads_started":1}
I20260927 15:07:53.047073 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.040676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:41521 (local address 127.0.0.1:41444)
0927 15:07:53.041699 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:53.041713 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:53.041798 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:53.042681 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:53.042684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:53.042714 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:53.042834 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.042841 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.044288 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.044291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.045428 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.045434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.046598 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.046601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.046747 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.046765 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 15:07:53.046815 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 15:07:53.046902 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"mutex_wait_us":30,"spinlock_wait_cycles":1536,"thread_start_us":104,"threads_started":1}
I20260927 15:07:53.047081 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.041429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:39663 (local address 127.0.0.1:36006)
0927 15:07:53.042195 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:53.042207 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:53.042281 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:53.042662 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:53.042665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:53.042682 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:53.042786 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.042792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.044156 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.044159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.045395 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.045406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.046574 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.046578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.046723 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.046746 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:07:53.046815 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 15:07:53.046903 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":51,"spinlock_wait_cycles":4480,"thread_start_us":81,"threads_started":1}
I20260927 15:07:53.048151 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.041500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.7.17.195:41521)
0927 15:07:53.042094 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.042106 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.042129 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:07:53.042319 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.042325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.042522 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.042992 (+ 470us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:53.043120 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.044184 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.045541 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.046493 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.047707 (+ 1214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.047864 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:53.047913 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 15:07:53.048004 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":90,"threads_started":1}
I20260927 15:07:53.049696 15042 tablet_service.cc:1467] Processing CreateTablet for tablet f1e93262ba4d4300862d71b2f2cbeb30 (DEFAULT_TABLE table=test-table [id=1b0fa0db8b5241ada6d7b86e722301fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:53.049693 14902 tablet_service.cc:1467] Processing CreateTablet for tablet f1e93262ba4d4300862d71b2f2cbeb30 (DEFAULT_TABLE table=test-table [id=1b0fa0db8b5241ada6d7b86e722301fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:53.050307 15182 tablet_service.cc:1467] Processing CreateTablet for tablet f1e93262ba4d4300862d71b2f2cbeb30 (DEFAULT_TABLE table=test-table [id=1b0fa0db8b5241ada6d7b86e722301fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:53.050815 15042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1e93262ba4d4300862d71b2f2cbeb30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:53.050815 14902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1e93262ba4d4300862d71b2f2cbeb30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:53.051208 15182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1e93262ba4d4300862d71b2f2cbeb30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:53.058212 15276 tablet_bootstrap.cc:492] T f1e93262ba4d4300862d71b2f2cbeb30 P 5ef67695d9fb4e979f286cb81e737490: Bootstrap starting.
I20260927 15:07:53.058874 15278 tablet_bootstrap.cc:492] T f1e93262ba4d4300862d71b2f2cbeb30 P 6b1c2b30413046bbb45b6d321ddab1a2: Bootstrap starting.
I20260927 15:07:53.059520 15277 tablet_bootstrap.cc:492] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e: Bootstrap starting.
I20260927 15:07:53.060253 15276 tablet_bootstrap.cc:654] T f1e93262ba4d4300862d71b2f2cbeb30 P 5ef67695d9fb4e979f286cb81e737490: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:53.061002 15276 log.cc:824] T f1e93262ba4d4300862d71b2f2cbeb30 P 5ef67695d9fb4e979f286cb81e737490: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:53.061050 15278 tablet_bootstrap.cc:654] T f1e93262ba4d4300862d71b2f2cbeb30 P 6b1c2b30413046bbb45b6d321ddab1a2: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:53.061596 15277 tablet_bootstrap.cc:654] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:53.061857 15278 log.cc:824] T f1e93262ba4d4300862d71b2f2cbeb30 P 6b1c2b30413046bbb45b6d321ddab1a2: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:53.062858 15277 log.cc:824] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:53.063277 15276 tablet_bootstrap.cc:492] T f1e93262ba4d4300862d71b2f2cbeb30 P 5ef67695d9fb4e979f286cb81e737490: No bootstrap required, opened a new log
I20260927 15:07:53.063462 15276 ts_tablet_manager.cc:1397] T f1e93262ba4d4300862d71b2f2cbeb30 P 5ef67695d9fb4e979f286cb81e737490: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:07:53.063828 15278 tablet_bootstrap.cc:492] T f1e93262ba4d4300862d71b2f2cbeb30 P 6b1c2b30413046bbb45b6d321ddab1a2: No bootstrap required, opened a new log
I20260927 15:07:53.063961 15278 ts_tablet_manager.cc:1397] T f1e93262ba4d4300862d71b2f2cbeb30 P 6b1c2b30413046bbb45b6d321ddab1a2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:07:53.064736 15277 tablet_bootstrap.cc:492] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e: No bootstrap required, opened a new log
I20260927 15:07:53.064945 15277 ts_tablet_manager.cc:1397] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:07:53.066939 15278 raft_consensus.cc:348] T f1e93262ba4d4300862d71b2f2cbeb30 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } }
I20260927 15:07:53.067162 15278 raft_consensus.cc:374] T f1e93262ba4d4300862d71b2f2cbeb30 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:53.067214 15278 raft_consensus.cc:729] T f1e93262ba4d4300862d71b2f2cbeb30 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1c2b30413046bbb45b6d321ddab1a2, State: Initialized, Role: FOLLOWER
I20260927 15:07:53.067456 15276 raft_consensus.cc:348] T f1e93262ba4d4300862d71b2f2cbeb30 P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } }
I20260927 15:07:53.067629 15278 consensus_queue.cc:260] T f1e93262ba4d4300862d71b2f2cbeb30 P 6b1c2b30413046bbb45b6d321ddab1a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } }
I20260927 15:07:53.067703 15276 raft_consensus.cc:374] T f1e93262ba4d4300862d71b2f2cbeb30 P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:53.067760 15276 raft_consensus.cc:729] T f1e93262ba4d4300862d71b2f2cbeb30 P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ef67695d9fb4e979f286cb81e737490, State: Initialized, Role: FOLLOWER
I20260927 15:07:53.068281 15108 heartbeater.cc:503] Master 127.7.17.254:40831 was elected leader, sending a full tablet report...
I20260927 15:07:53.068269 15276 consensus_queue.cc:260] T f1e93262ba4d4300862d71b2f2cbeb30 P 5ef67695d9fb4e979f286cb81e737490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } }
I20260927 15:07:53.069334 15276 ts_tablet_manager.cc:1428] T f1e93262ba4d4300862d71b2f2cbeb30 P 5ef67695d9fb4e979f286cb81e737490: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:07:53.069306 15277 raft_consensus.cc:348] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } }
I20260927 15:07:53.069558 15277 raft_consensus.cc:374] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:53.069603 15277 raft_consensus.cc:729] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36a116f98c2d4cb8b337e3a71a180e8e, State: Initialized, Role: FOLLOWER
I20260927 15:07:53.069687 15278 ts_tablet_manager.cc:1428] T f1e93262ba4d4300862d71b2f2cbeb30 P 6b1c2b30413046bbb45b6d321ddab1a2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:07:53.069751 15248 heartbeater.cc:503] Master 127.7.17.254:40831 was elected leader, sending a full tablet report...
I20260927 15:07:53.070036 15277 consensus_queue.cc:260] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } }
I20260927 15:07:53.070640 14968 heartbeater.cc:503] Master 127.7.17.254:40831 was elected leader, sending a full tablet report...
I20260927 15:07:53.071149 15277 ts_tablet_manager.cc:1428] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 15:07:53.226496 15249 tablet.cc:2369] T f1e93262ba4d4300862d71b2f2cbeb30 P 5ef67695d9fb4e979f286cb81e737490: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:07:53.260619 15109 tablet.cc:2369] T f1e93262ba4d4300862d71b2f2cbeb30 P 6b1c2b30413046bbb45b6d321ddab1a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:07:53.284914 15284 raft_consensus.cc:482] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:53.285192 15284 raft_consensus.cc:504] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } }
I20260927 15:07:53.286324 15284 leader_election.cc:288] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ef67695d9fb4e979f286cb81e737490 (127.7.17.195:41521), 6b1c2b30413046bbb45b6d321ddab1a2 (127.7.17.194:39663)
I20260927 15:07:53.290223 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.286686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:58531 (local address 127.7.17.195:41521)
0927 15:07:53.286796 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.286801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.287042 (+ 241us) server_negotiation.cc:407] Connection header received
0927 15:07:53.287175 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.287178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.287223 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.287282 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:53.287585 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.288079 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.289264 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.289654 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.289761 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.289816 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:53.290022 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 15:07:53.290092 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:07:53.290227 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.286654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:45259 (local address 127.7.17.194:39663)
0927 15:07:53.286779 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.286783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.287126 (+ 343us) server_negotiation.cc:407] Connection header received
0927 15:07:53.287185 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.287188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.287230 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.287292 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:53.287583 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.288087 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.289268 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.289658 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.289761 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.289816 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:53.290027 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 15:07:53.290110 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:07:53.290251 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.286554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:41521 (local address 127.7.17.193:58531)
0927 15:07:53.286933 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:53.286958 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:53.287075 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:53.287299 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:53.287301 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:53.287315 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:53.287486 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.287491 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.288179 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.288183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.289153 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.289163 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.289761 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.289765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.289876 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.289929 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 15:07:53.289997 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 15:07:53.290085 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":78,"spinlock_wait_cycles":8064,"thread_start_us":116,"threads_started":1}
I20260927 15:07:53.290257 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.286563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:39663 (local address 127.7.17.193:45259)
0927 15:07:53.286977 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:53.286988 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:53.287088 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:53.287307 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:53.287310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:53.287324 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:53.287479 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.287485 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.288182 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.288186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.289157 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.289166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.289761 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.289765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.289914 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.289936 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:07:53.289998 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 15:07:53.290093 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":309,"thread_start_us":66,"threads_started":1}
I20260927 15:07:53.290961 15062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e93262ba4d4300862d71b2f2cbeb30" candidate_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" is_pre_election: true
I20260927 15:07:53.290980 15202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e93262ba4d4300862d71b2f2cbeb30" candidate_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ef67695d9fb4e979f286cb81e737490" is_pre_election: true
I20260927 15:07:53.291244 15202 raft_consensus.cc:2454] T f1e93262ba4d4300862d71b2f2cbeb30 P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36a116f98c2d4cb8b337e3a71a180e8e in term 0.
I20260927 15:07:53.291257 15062 raft_consensus.cc:2454] T f1e93262ba4d4300862d71b2f2cbeb30 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36a116f98c2d4cb8b337e3a71a180e8e in term 0.
I20260927 15:07:53.291689 14856 leader_election.cc:302] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36a116f98c2d4cb8b337e3a71a180e8e, 6b1c2b30413046bbb45b6d321ddab1a2; no voters:
I20260927 15:07:53.291960 15284 raft_consensus.cc:2781] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:53.292037 15284 raft_consensus.cc:482] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:53.292068 15284 raft_consensus.cc:3037] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Advancing to term 1
W20260927 15:07:53.292470 14969 tablet.cc:2369] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:07:53.293241 15284 raft_consensus.cc:504] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } }
I20260927 15:07:53.293581 15284 leader_election.cc:288] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e [CANDIDATE]: Term 1 election: Requested vote from peers 5ef67695d9fb4e979f286cb81e737490 (127.7.17.195:41521), 6b1c2b30413046bbb45b6d321ddab1a2 (127.7.17.194:39663)
I20260927 15:07:53.293977 15062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e93262ba4d4300862d71b2f2cbeb30" candidate_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1c2b30413046bbb45b6d321ddab1a2"
I20260927 15:07:53.293967 15202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e93262ba4d4300862d71b2f2cbeb30" candidate_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ef67695d9fb4e979f286cb81e737490"
I20260927 15:07:53.294112 15202 raft_consensus.cc:3037] T f1e93262ba4d4300862d71b2f2cbeb30 P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:53.294106 15062 raft_consensus.cc:3037] T f1e93262ba4d4300862d71b2f2cbeb30 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:53.295385 15062 raft_consensus.cc:2454] T f1e93262ba4d4300862d71b2f2cbeb30 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36a116f98c2d4cb8b337e3a71a180e8e in term 1.
I20260927 15:07:53.295720 14856 leader_election.cc:302] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36a116f98c2d4cb8b337e3a71a180e8e, 6b1c2b30413046bbb45b6d321ddab1a2; no voters:
I20260927 15:07:53.295832 15202 raft_consensus.cc:2454] T f1e93262ba4d4300862d71b2f2cbeb30 P 5ef67695d9fb4e979f286cb81e737490 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36a116f98c2d4cb8b337e3a71a180e8e in term 1.
I20260927 15:07:53.295881 15284 raft_consensus.cc:2781] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:53.296113 15284 raft_consensus.cc:686] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e [term 1 LEADER]: Becoming Leader. State: Replica: 36a116f98c2d4cb8b337e3a71a180e8e, State: Running, Role: LEADER
I20260927 15:07:53.296465 15284 consensus_queue.cc:237] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } }
I20260927 15:07:53.300360 14765 catalog_manager.cc:5495] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 36a116f98c2d4cb8b337e3a71a180e8e (127.7.17.193). New cstate: current_term: 1 leader_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } health_report { overall_health: UNKNOWN } } }
I20260927 15:07:53.346873 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.343249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54944 (local address 127.7.17.193:41869)
0927 15:07:53.343372 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.343377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.343392 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:07:53.343562 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.343565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.343627 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.343721 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:53.343957 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.344449 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.345424 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.345646 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.345732 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.345996 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:53.346479 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:53.346590 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:53.346672 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 15:07:53.346732 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:07:53.352800 15202 raft_consensus.cc:1261] T f1e93262ba4d4300862d71b2f2cbeb30 P 5ef67695d9fb4e979f286cb81e737490 [term 1 FOLLOWER]: Refusing update from remote peer 36a116f98c2d4cb8b337e3a71a180e8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:53.352800 15062 raft_consensus.cc:1261] T f1e93262ba4d4300862d71b2f2cbeb30 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 1 FOLLOWER]: Refusing update from remote peer 36a116f98c2d4cb8b337e3a71a180e8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:53.353531 15284 consensus_queue.cc:1035] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:53.353757 15287 consensus_queue.cc:1035] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:53.361188 15291 mvcc.cc:205] Tried to move back new op lower bound from 7333976774046175232 to 7333976773825531904. Current Snapshot: MvccSnapshot[applied={T|T < 7333976774046175232}]
I20260927 15:07:53.363096 15292 mvcc.cc:205] Tried to move back new op lower bound from 7333976774046175232 to 7333976773825531904. Current Snapshot: MvccSnapshot[applied={T|T < 7333976774046175232}]
I20260927 15:07:53.370132 14764 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:53.370312 14764 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:53.372792 14764 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9: Sending DeleteTablet for 3 replicas of tablet f1e93262ba4d4300862d71b2f2cbeb30
I20260927 15:07:53.373713 14902 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e93262ba4d4300862d71b2f2cbeb30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:53 UTC) from {username='oryx', principal='oryx/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:54940
I20260927 15:07:53.373719 15182 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e93262ba4d4300862d71b2f2cbeb30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:53 UTC) from {username='oryx', principal='oryx/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:41444
I20260927 15:07:53.374296 15300 tablet_replica.cc:330] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e: stopping tablet replica
I20260927 15:07:53.374265 15042 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e93262ba4d4300862d71b2f2cbeb30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:53 UTC) from {username='oryx', principal='oryx/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:36006
I20260927 15:07:53.374675 15300 raft_consensus.cc:2229] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:07:53.374707 15301 tablet_replica.cc:330] T f1e93262ba4d4300862d71b2f2cbeb30 P 6b1c2b30413046bbb45b6d321ddab1a2: stopping tablet replica
I20260927 15:07:53.375073 15300 raft_consensus.cc:2258] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:53.375221 15301 raft_consensus.cc:2229] T f1e93262ba4d4300862d71b2f2cbeb30 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:53.375466 15299 tablet_replica.cc:330] T f1e93262ba4d4300862d71b2f2cbeb30 P 5ef67695d9fb4e979f286cb81e737490: stopping tablet replica
I20260927 15:07:53.375669 15301 raft_consensus.cc:2258] T f1e93262ba4d4300862d71b2f2cbeb30 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:53.375772 15299 raft_consensus.cc:2229] T f1e93262ba4d4300862d71b2f2cbeb30 P 5ef67695d9fb4e979f286cb81e737490 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:53.375756 14764 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 15:07:53.376068 15299 raft_consensus.cc:2258] T f1e93262ba4d4300862d71b2f2cbeb30 P 5ef67695d9fb4e979f286cb81e737490 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 15:07:53.376206 14764 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:53.378257 15301 ts_tablet_manager.cc:1905] T f1e93262ba4d4300862d71b2f2cbeb30 P 6b1c2b30413046bbb45b6d321ddab1a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:53.378258 15300 ts_tablet_manager.cc:1905] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:53.381183 15300 ts_tablet_manager.cc:1918] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:53.381291 15300 log.cc:1196] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-0/wal/wals/f1e93262ba4d4300862d71b2f2cbeb30
I20260927 15:07:53.381601 15300 ts_tablet_manager.cc:1939] T f1e93262ba4d4300862d71b2f2cbeb30 P 36a116f98c2d4cb8b337e3a71a180e8e: Deleting consensus metadata
I20260927 15:07:53.381731 15182 tablet_service.cc:1467] Processing CreateTablet for tablet dccb5cc944c44a33b82ed5242bda8b33 (DEFAULT_TABLE table=test-table [id=c3503f76cc844d8a990fab96b579d793]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:53.381821 15042 tablet_service.cc:1467] Processing CreateTablet for tablet dccb5cc944c44a33b82ed5242bda8b33 (DEFAULT_TABLE table=test-table [id=c3503f76cc844d8a990fab96b579d793]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:53.382011 15182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dccb5cc944c44a33b82ed5242bda8b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:53.382207 14902 tablet_service.cc:1467] Processing CreateTablet for tablet dccb5cc944c44a33b82ed5242bda8b33 (DEFAULT_TABLE table=test-table [id=c3503f76cc844d8a990fab96b579d793]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:53.382432 14902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dccb5cc944c44a33b82ed5242bda8b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:53.384202 15299 ts_tablet_manager.cc:1905] T f1e93262ba4d4300862d71b2f2cbeb30 P 5ef67695d9fb4e979f286cb81e737490: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:53.384538 15277 tablet_bootstrap.cc:492] T dccb5cc944c44a33b82ed5242bda8b33 P 36a116f98c2d4cb8b337e3a71a180e8e: Bootstrap starting.
I20260927 15:07:53.384583 15276 tablet_bootstrap.cc:492] T dccb5cc944c44a33b82ed5242bda8b33 P 5ef67695d9fb4e979f286cb81e737490: Bootstrap starting.
I20260927 15:07:53.385485 15277 tablet_bootstrap.cc:654] T dccb5cc944c44a33b82ed5242bda8b33 P 36a116f98c2d4cb8b337e3a71a180e8e: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:53.386025 15276 tablet_bootstrap.cc:654] T dccb5cc944c44a33b82ed5242bda8b33 P 5ef67695d9fb4e979f286cb81e737490: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:53.386201 14752 catalog_manager.cc:4841] TS 36a116f98c2d4cb8b337e3a71a180e8e (127.7.17.193:41869): tablet f1e93262ba4d4300862d71b2f2cbeb30 (table test-table [id=1b0fa0db8b5241ada6d7b86e722301fe]) successfully deleted
I20260927 15:07:53.386523 15299 ts_tablet_manager.cc:1918] T f1e93262ba4d4300862d71b2f2cbeb30 P 5ef67695d9fb4e979f286cb81e737490: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:53.386605 15299 log.cc:1196] T f1e93262ba4d4300862d71b2f2cbeb30 P 5ef67695d9fb4e979f286cb81e737490: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/f1e93262ba4d4300862d71b2f2cbeb30
I20260927 15:07:53.386808 15042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dccb5cc944c44a33b82ed5242bda8b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:53.386873 15299 ts_tablet_manager.cc:1939] T f1e93262ba4d4300862d71b2f2cbeb30 P 5ef67695d9fb4e979f286cb81e737490: Deleting consensus metadata
I20260927 15:07:53.388150 15301 ts_tablet_manager.cc:1918] T f1e93262ba4d4300862d71b2f2cbeb30 P 6b1c2b30413046bbb45b6d321ddab1a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:53.388126 14749 catalog_manager.cc:4841] TS 5ef67695d9fb4e979f286cb81e737490 (127.7.17.195:41521): tablet f1e93262ba4d4300862d71b2f2cbeb30 (table test-table [id=1b0fa0db8b5241ada6d7b86e722301fe]) successfully deleted
I20260927 15:07:53.388269 15301 log.cc:1196] T f1e93262ba4d4300862d71b2f2cbeb30 P 6b1c2b30413046bbb45b6d321ddab1a2: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-1/wal/wals/f1e93262ba4d4300862d71b2f2cbeb30
I20260927 15:07:53.388564 15301 ts_tablet_manager.cc:1939] T f1e93262ba4d4300862d71b2f2cbeb30 P 6b1c2b30413046bbb45b6d321ddab1a2: Deleting consensus metadata
I20260927 15:07:53.389283 15278 tablet_bootstrap.cc:492] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2: Bootstrap starting.
I20260927 15:07:53.389880 14752 catalog_manager.cc:4841] TS 6b1c2b30413046bbb45b6d321ddab1a2 (127.7.17.194:39663): tablet f1e93262ba4d4300862d71b2f2cbeb30 (table test-table [id=1b0fa0db8b5241ada6d7b86e722301fe]) successfully deleted
I20260927 15:07:53.390200 15278 tablet_bootstrap.cc:654] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:53.390726 15277 tablet_bootstrap.cc:492] T dccb5cc944c44a33b82ed5242bda8b33 P 36a116f98c2d4cb8b337e3a71a180e8e: No bootstrap required, opened a new log
I20260927 15:07:53.390761 15276 tablet_bootstrap.cc:492] T dccb5cc944c44a33b82ed5242bda8b33 P 5ef67695d9fb4e979f286cb81e737490: No bootstrap required, opened a new log
I20260927 15:07:53.390816 15277 ts_tablet_manager.cc:1397] T dccb5cc944c44a33b82ed5242bda8b33 P 36a116f98c2d4cb8b337e3a71a180e8e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 15:07:53.390838 15276 ts_tablet_manager.cc:1397] T dccb5cc944c44a33b82ed5242bda8b33 P 5ef67695d9fb4e979f286cb81e737490: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260927 15:07:53.391276 15276 raft_consensus.cc:348] T dccb5cc944c44a33b82ed5242bda8b33 P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.391294 15277 raft_consensus.cc:348] T dccb5cc944c44a33b82ed5242bda8b33 P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.391390 15276 raft_consensus.cc:374] T dccb5cc944c44a33b82ed5242bda8b33 P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:53.391395 15277 raft_consensus.cc:374] T dccb5cc944c44a33b82ed5242bda8b33 P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:53.391422 15276 raft_consensus.cc:729] T dccb5cc944c44a33b82ed5242bda8b33 P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ef67695d9fb4e979f286cb81e737490, State: Initialized, Role: FOLLOWER
I20260927 15:07:53.391424 15277 raft_consensus.cc:729] T dccb5cc944c44a33b82ed5242bda8b33 P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36a116f98c2d4cb8b337e3a71a180e8e, State: Initialized, Role: FOLLOWER
I20260927 15:07:53.391557 15277 consensus_queue.cc:260] T dccb5cc944c44a33b82ed5242bda8b33 P 36a116f98c2d4cb8b337e3a71a180e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.391554 15276 consensus_queue.cc:260] T dccb5cc944c44a33b82ed5242bda8b33 P 5ef67695d9fb4e979f286cb81e737490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.391690 15278 tablet_bootstrap.cc:492] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2: No bootstrap required, opened a new log
I20260927 15:07:53.391758 15278 ts_tablet_manager.cc:1397] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 15:07:53.391795 15277 ts_tablet_manager.cc:1428] T dccb5cc944c44a33b82ed5242bda8b33 P 36a116f98c2d4cb8b337e3a71a180e8e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:53.391809 15276 ts_tablet_manager.cc:1428] T dccb5cc944c44a33b82ed5242bda8b33 P 5ef67695d9fb4e979f286cb81e737490: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 15:07:53.392198 15278 raft_consensus.cc:348] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.392304 15278 raft_consensus.cc:374] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:53.392333 15278 raft_consensus.cc:729] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1c2b30413046bbb45b6d321ddab1a2, State: Initialized, Role: FOLLOWER
I20260927 15:07:53.392441 15278 consensus_queue.cc:260] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.392668 15278 ts_tablet_manager.cc:1428] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:53.585059 15282 raft_consensus.cc:482] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:53.585217 15282 raft_consensus.cc:504] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.586480 15282 leader_election.cc:288] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36a116f98c2d4cb8b337e3a71a180e8e (127.7.17.193:41869), 5ef67695d9fb4e979f286cb81e737490 (127.7.17.195:41521)
I20260927 15:07:53.591989 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.586847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:50767 (local address 127.7.17.193:41869)
0927 15:07:53.586989 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.586994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.587171 (+ 177us) server_negotiation.cc:407] Connection header received
0927 15:07:53.587311 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.587315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.587362 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.587445 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:53.587854 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.588662 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.590600 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.591241 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.591375 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.591433 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:53.591761 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 15:07:53.591851 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:07:53.591989 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.587135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:40535 (local address 127.7.17.195:41521)
0927 15:07:53.587255 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.587259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.587272 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:07:53.587316 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.587319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.587362 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.587445 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:53.587855 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.588665 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.590600 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.591242 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.591375 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.591434 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:53.591761 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 15:07:53.591851 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:07:53.592131 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.586750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:41521 (local address 127.7.17.194:40535)
0927 15:07:53.587066 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:53.587081 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:53.587201 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:53.587468 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:53.587472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:53.587521 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:53.587734 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.587741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.588809 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.588813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.590435 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.590448 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.591416 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.591420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.591585 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.591620 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 15:07:53.591704 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 15:07:53.591817 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":159,"spinlock_wait_cycles":15744,"thread_start_us":112,"threads_started":1}
I20260927 15:07:53.592442 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.586712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:41869 (local address 127.7.17.194:50767)
0927 15:07:53.587066 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:53.587081 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:53.587201 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:53.587468 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:53.587472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:53.587521 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:53.587735 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.587741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.588809 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.588813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.590435 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.590448 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.591416 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.591420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.591569 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.591626 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 15:07:53.591704 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 15:07:53.591826 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":246,"spinlock_wait_cycles":89472,"thread_start_us":101,"threads_started":1}
I20260927 15:07:53.592641 15202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dccb5cc944c44a33b82ed5242bda8b33" candidate_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ef67695d9fb4e979f286cb81e737490" is_pre_election: true
I20260927 15:07:53.592814 15202 raft_consensus.cc:2454] T dccb5cc944c44a33b82ed5242bda8b33 P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b1c2b30413046bbb45b6d321ddab1a2 in term 0.
I20260927 15:07:53.593012 14922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dccb5cc944c44a33b82ed5242bda8b33" candidate_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" is_pre_election: true
I20260927 15:07:53.593261 14993 leader_election.cc:302] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ef67695d9fb4e979f286cb81e737490, 6b1c2b30413046bbb45b6d321ddab1a2; no voters:
I20260927 15:07:53.593317 14922 raft_consensus.cc:2454] T dccb5cc944c44a33b82ed5242bda8b33 P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b1c2b30413046bbb45b6d321ddab1a2 in term 0.
I20260927 15:07:53.593539 15282 raft_consensus.cc:2781] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:53.593600 15282 raft_consensus.cc:482] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:53.593638 15282 raft_consensus.cc:3037] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:53.594614 15282 raft_consensus.cc:504] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.594947 15282 leader_election.cc:288] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2 [CANDIDATE]: Term 1 election: Requested vote from peers 36a116f98c2d4cb8b337e3a71a180e8e (127.7.17.193:41869), 5ef67695d9fb4e979f286cb81e737490 (127.7.17.195:41521)
I20260927 15:07:53.595295 15202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dccb5cc944c44a33b82ed5242bda8b33" candidate_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ef67695d9fb4e979f286cb81e737490"
I20260927 15:07:53.595295 14922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dccb5cc944c44a33b82ed5242bda8b33" candidate_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36a116f98c2d4cb8b337e3a71a180e8e"
I20260927 15:07:53.595418 14922 raft_consensus.cc:3037] T dccb5cc944c44a33b82ed5242bda8b33 P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:53.595418 15202 raft_consensus.cc:3037] T dccb5cc944c44a33b82ed5242bda8b33 P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:53.596498 14922 raft_consensus.cc:2454] T dccb5cc944c44a33b82ed5242bda8b33 P 36a116f98c2d4cb8b337e3a71a180e8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b1c2b30413046bbb45b6d321ddab1a2 in term 1.
I20260927 15:07:53.596498 15202 raft_consensus.cc:2454] T dccb5cc944c44a33b82ed5242bda8b33 P 5ef67695d9fb4e979f286cb81e737490 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b1c2b30413046bbb45b6d321ddab1a2 in term 1.
I20260927 15:07:53.596809 14996 leader_election.cc:302] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36a116f98c2d4cb8b337e3a71a180e8e, 6b1c2b30413046bbb45b6d321ddab1a2; no voters:
I20260927 15:07:53.597110 15282 raft_consensus.cc:2781] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:53.597322 15282 raft_consensus.cc:686] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 1 LEADER]: Becoming Leader. State: Replica: 6b1c2b30413046bbb45b6d321ddab1a2, State: Running, Role: LEADER
I20260927 15:07:53.597591 15282 consensus_queue.cc:237] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.599867 14764 catalog_manager.cc:5495] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b1c2b30413046bbb45b6d321ddab1a2 (127.7.17.194). New cstate: current_term: 1 leader_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } health_report { overall_health: UNKNOWN } } }
Sep 27 15:07:53 dist-test-slave-hhpq krb5kdc[14720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521672, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.254@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM
I20260927 15:07:53.667647 15314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.654596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:40831 (local address 127.0.0.1:46654)
0927 15:07:53.655020 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:53.655036 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:53.655149 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:53.655414 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:53.655416 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:53.655835 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:53.655929 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.655935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.656728 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.656730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.657360 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.657370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.657848 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.657850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.657915 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.662505 (+ 4590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:53.662520 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:53.664203 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:53.666030 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:53.666043 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:53.666051 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:53.666300 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:53.666577 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:53.666579 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:53.666580 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:53.666651 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:53.667029 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:53.667038 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:53.667152 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 15:07:53.667248 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 15:07:53.667422 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":136,"threads_started":1}
I20260927 15:07:53.667647 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.654653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46654 (local address 127.7.17.254:40831)
0927 15:07:53.654940 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.654948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.655112 (+ 164us) server_negotiation.cc:407] Connection header received
0927 15:07:53.655248 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.655250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.655294 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.655385 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:53.656069 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.656633 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.657515 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.657743 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.657824 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.664428 (+ 6604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:53.664432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:53.664434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:53.664460 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:53.665917 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:53.666392 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:53.666396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:53.666398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:53.666475 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:53.666728 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:53.666732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:53.666734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:53.666927 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:53.667032 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:53.667313 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 15:07:53.667478 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":210,"thread_start_us":91,"threads_started":1}
I20260927 15:07:53.671892 14764 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:46654:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 15:07:53.672674 14764 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:53.677790 15042 tablet_service.cc:1467] Processing CreateTablet for tablet d7066e40fcd94444ae02b4dfb2876b36 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e3d803a6448b4882b0468bef3f2b85cf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:07:53.677784 14902 tablet_service.cc:1467] Processing CreateTablet for tablet d7066e40fcd94444ae02b4dfb2876b36 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e3d803a6448b4882b0468bef3f2b85cf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:07:53.677785 15182 tablet_service.cc:1467] Processing CreateTablet for tablet d7066e40fcd94444ae02b4dfb2876b36 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e3d803a6448b4882b0468bef3f2b85cf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:07:53.678172 14902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7066e40fcd94444ae02b4dfb2876b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:53.678172 15182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7066e40fcd94444ae02b4dfb2876b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:53.678179 15042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7066e40fcd94444ae02b4dfb2876b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:53.680814 15278 tablet_bootstrap.cc:492] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2: Bootstrap starting.
I20260927 15:07:53.680897 15277 tablet_bootstrap.cc:492] T d7066e40fcd94444ae02b4dfb2876b36 P 36a116f98c2d4cb8b337e3a71a180e8e: Bootstrap starting.
I20260927 15:07:53.681643 15277 tablet_bootstrap.cc:654] T d7066e40fcd94444ae02b4dfb2876b36 P 36a116f98c2d4cb8b337e3a71a180e8e: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:53.681680 15278 tablet_bootstrap.cc:654] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:53.682269 15276 tablet_bootstrap.cc:492] T d7066e40fcd94444ae02b4dfb2876b36 P 5ef67695d9fb4e979f286cb81e737490: Bootstrap starting.
I20260927 15:07:53.682730 15277 tablet_bootstrap.cc:492] T d7066e40fcd94444ae02b4dfb2876b36 P 36a116f98c2d4cb8b337e3a71a180e8e: No bootstrap required, opened a new log
I20260927 15:07:53.682808 15277 ts_tablet_manager.cc:1397] T d7066e40fcd94444ae02b4dfb2876b36 P 36a116f98c2d4cb8b337e3a71a180e8e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 15:07:53.682844 15278 tablet_bootstrap.cc:492] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2: No bootstrap required, opened a new log
I20260927 15:07:53.682931 15278 ts_tablet_manager.cc:1397] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:07:53.683197 15276 tablet_bootstrap.cc:654] T d7066e40fcd94444ae02b4dfb2876b36 P 5ef67695d9fb4e979f286cb81e737490: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:53.683228 15277 raft_consensus.cc:348] T d7066e40fcd94444ae02b4dfb2876b36 P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.683350 15277 raft_consensus.cc:374] T d7066e40fcd94444ae02b4dfb2876b36 P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:53.683384 15277 raft_consensus.cc:729] T d7066e40fcd94444ae02b4dfb2876b36 P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36a116f98c2d4cb8b337e3a71a180e8e, State: Initialized, Role: FOLLOWER
I20260927 15:07:53.683385 15278 raft_consensus.cc:348] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.683544 15278 raft_consensus.cc:374] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:53.683509 15277 consensus_queue.cc:260] T d7066e40fcd94444ae02b4dfb2876b36 P 36a116f98c2d4cb8b337e3a71a180e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.683584 15278 raft_consensus.cc:729] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1c2b30413046bbb45b6d321ddab1a2, State: Initialized, Role: FOLLOWER
I20260927 15:07:53.683703 15278 consensus_queue.cc:260] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.683780 15277 ts_tablet_manager.cc:1428] T d7066e40fcd94444ae02b4dfb2876b36 P 36a116f98c2d4cb8b337e3a71a180e8e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 15:07:53.683954 15278 ts_tablet_manager.cc:1428] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:53.683954 15287 tablet_replica.cc:439] T d7066e40fcd94444ae02b4dfb2876b36 P 36a116f98c2d4cb8b337e3a71a180e8e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } } }
I20260927 15:07:53.684095 15287 tablet_replica.cc:442] T d7066e40fcd94444ae02b4dfb2876b36 P 36a116f98c2d4cb8b337e3a71a180e8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:53.684544 15282 tablet_replica.cc:439] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } } }
I20260927 15:07:53.684684 15282 tablet_replica.cc:442] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:53.685060 15276 tablet_bootstrap.cc:492] T d7066e40fcd94444ae02b4dfb2876b36 P 5ef67695d9fb4e979f286cb81e737490: No bootstrap required, opened a new log
I20260927 15:07:53.685156 15276 ts_tablet_manager.cc:1397] T d7066e40fcd94444ae02b4dfb2876b36 P 5ef67695d9fb4e979f286cb81e737490: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 15:07:53.685616 15276 raft_consensus.cc:348] T d7066e40fcd94444ae02b4dfb2876b36 P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.685735 15276 raft_consensus.cc:374] T d7066e40fcd94444ae02b4dfb2876b36 P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:53.685776 15276 raft_consensus.cc:729] T d7066e40fcd94444ae02b4dfb2876b36 P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ef67695d9fb4e979f286cb81e737490, State: Initialized, Role: FOLLOWER
I20260927 15:07:53.685899 15276 consensus_queue.cc:260] T d7066e40fcd94444ae02b4dfb2876b36 P 5ef67695d9fb4e979f286cb81e737490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.686172 15276 ts_tablet_manager.cc:1428] T d7066e40fcd94444ae02b4dfb2876b36 P 5ef67695d9fb4e979f286cb81e737490: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:53.686362 15283 tablet_replica.cc:439] T d7066e40fcd94444ae02b4dfb2876b36 P 5ef67695d9fb4e979f286cb81e737490: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } } }
I20260927 15:07:53.686503 15283 tablet_replica.cc:442] T d7066e40fcd94444ae02b4dfb2876b36 P 5ef67695d9fb4e979f286cb81e737490: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:53.735545 15282 raft_consensus.cc:482] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:53.735682 15282 raft_consensus.cc:504] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.736104 15282 leader_election.cc:288] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36a116f98c2d4cb8b337e3a71a180e8e (127.7.17.193:41869), 5ef67695d9fb4e979f286cb81e737490 (127.7.17.195:41521)
I20260927 15:07:53.736565 14922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7066e40fcd94444ae02b4dfb2876b36" candidate_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" is_pre_election: true
I20260927 15:07:53.736619 15202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7066e40fcd94444ae02b4dfb2876b36" candidate_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ef67695d9fb4e979f286cb81e737490" is_pre_election: true
I20260927 15:07:53.736723 14922 raft_consensus.cc:2454] T d7066e40fcd94444ae02b4dfb2876b36 P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b1c2b30413046bbb45b6d321ddab1a2 in term 0.
I20260927 15:07:53.736747 15202 raft_consensus.cc:2454] T d7066e40fcd94444ae02b4dfb2876b36 P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b1c2b30413046bbb45b6d321ddab1a2 in term 0.
I20260927 15:07:53.737069 14993 leader_election.cc:302] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 36a116f98c2d4cb8b337e3a71a180e8e, 5ef67695d9fb4e979f286cb81e737490, 6b1c2b30413046bbb45b6d321ddab1a2; no voters:
I20260927 15:07:53.737253 15282 raft_consensus.cc:2781] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:53.737314 15282 raft_consensus.cc:482] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:53.737352 15282 raft_consensus.cc:3037] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:53.738329 15282 raft_consensus.cc:504] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.738649 15282 leader_election.cc:288] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2 [CANDIDATE]: Term 1 election: Requested vote from peers 36a116f98c2d4cb8b337e3a71a180e8e (127.7.17.193:41869), 5ef67695d9fb4e979f286cb81e737490 (127.7.17.195:41521)
I20260927 15:07:53.739017 14922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7066e40fcd94444ae02b4dfb2876b36" candidate_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36a116f98c2d4cb8b337e3a71a180e8e"
I20260927 15:07:53.739081 15202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7066e40fcd94444ae02b4dfb2876b36" candidate_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ef67695d9fb4e979f286cb81e737490"
I20260927 15:07:53.739174 14922 raft_consensus.cc:3037] T d7066e40fcd94444ae02b4dfb2876b36 P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:53.739205 15202 raft_consensus.cc:3037] T d7066e40fcd94444ae02b4dfb2876b36 P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:53.740053 14922 raft_consensus.cc:2454] T d7066e40fcd94444ae02b4dfb2876b36 P 36a116f98c2d4cb8b337e3a71a180e8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b1c2b30413046bbb45b6d321ddab1a2 in term 1.
I20260927 15:07:53.740083 15202 raft_consensus.cc:2454] T d7066e40fcd94444ae02b4dfb2876b36 P 5ef67695d9fb4e979f286cb81e737490 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b1c2b30413046bbb45b6d321ddab1a2 in term 1.
I20260927 15:07:53.740317 14996 leader_election.cc:302] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36a116f98c2d4cb8b337e3a71a180e8e, 6b1c2b30413046bbb45b6d321ddab1a2; no voters:
I20260927 15:07:53.740471 15282 raft_consensus.cc:2781] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:53.740567 15282 raft_consensus.cc:686] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 1 LEADER]: Becoming Leader. State: Replica: 6b1c2b30413046bbb45b6d321ddab1a2, State: Running, Role: LEADER
I20260927 15:07:53.740698 15282 consensus_queue.cc:237] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.741426 15307 tablet_replica.cc:439] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2: TxnStatusTablet state changed. Reason: New leader 6b1c2b30413046bbb45b6d321ddab1a2. Latest consensus state: current_term: 1 leader_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } } }
I20260927 15:07:53.741570 15307 tablet_replica.cc:442] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:07:53.741879 15320 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 15:07:53.742035 15320 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 15:07:53.742785 14765 catalog_manager.cc:5495] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b1c2b30413046bbb45b6d321ddab1a2 (127.7.17.194). New cstate: current_term: 1 leader_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } health_report { overall_health: UNKNOWN } } }
I20260927 15:07:53.769212 15314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.764002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:39663 (local address 127.0.0.1:36008)
0927 15:07:53.764279 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:53.764296 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:53.764425 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:53.764967 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:53.764970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:53.764982 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:07:53.765088 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.765093 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.766183 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.766187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.767167 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.767178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.767830 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.767834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.767924 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.767980 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:07:53.768862 (+ 882us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:07:53.768878 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 15:07:53.768934 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 15:07:53.769050 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260927 15:07:53.769579 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.764111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36008 (local address 127.7.17.194:39663)
0927 15:07:53.764281 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.764285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.764536 (+ 251us) server_negotiation.cc:407] Connection header received
0927 15:07:53.764647 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.764651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.764710 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.764798 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:53.765236 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.766049 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.767304 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.767662 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.767745 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.768144 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:53.768682 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:53.768791 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:53.769397 (+ 606us) server_negotiation.cc:299] Negotiation successful
0927 15:07:53.769467 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 15:07:53.772392 15202 raft_consensus.cc:1261] T d7066e40fcd94444ae02b4dfb2876b36 P 5ef67695d9fb4e979f286cb81e737490 [term 1 FOLLOWER]: Refusing update from remote peer 6b1c2b30413046bbb45b6d321ddab1a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:53.772603 14922 raft_consensus.cc:1261] T d7066e40fcd94444ae02b4dfb2876b36 P 36a116f98c2d4cb8b337e3a71a180e8e [term 1 FOLLOWER]: Refusing update from remote peer 6b1c2b30413046bbb45b6d321ddab1a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:53.772904 15282 consensus_queue.cc:1035] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:53.773165 15307 consensus_queue.cc:1035] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:53.774657 15283 tablet_replica.cc:439] T d7066e40fcd94444ae02b4dfb2876b36 P 5ef67695d9fb4e979f286cb81e737490: TxnStatusTablet state changed. Reason: New leader 6b1c2b30413046bbb45b6d321ddab1a2. Latest consensus state: current_term: 1 leader_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } } }
I20260927 15:07:53.774852 15283 tablet_replica.cc:442] T d7066e40fcd94444ae02b4dfb2876b36 P 5ef67695d9fb4e979f286cb81e737490: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:53.774945 15287 tablet_replica.cc:439] T d7066e40fcd94444ae02b4dfb2876b36 P 36a116f98c2d4cb8b337e3a71a180e8e: TxnStatusTablet state changed. Reason: New leader 6b1c2b30413046bbb45b6d321ddab1a2. Latest consensus state: current_term: 1 leader_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } } }
I20260927 15:07:53.775035 15287 tablet_replica.cc:442] T d7066e40fcd94444ae02b4dfb2876b36 P 36a116f98c2d4cb8b337e3a71a180e8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:53.777557 15282 tablet_replica.cc:439] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } } }
I20260927 15:07:53.777678 15282 tablet_replica.cc:442] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:07:53.777576 15307 tablet_replica.cc:439] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } } }
I20260927 15:07:53.778148 15307 tablet_replica.cc:442] T d7066e40fcd94444ae02b4dfb2876b36 P 6b1c2b30413046bbb45b6d321ddab1a2: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:07:53.778869 15290 mvcc.cc:205] Tried to move back new op lower bound from 7333976775766757376 to 7333976775643475968. Current Snapshot: MvccSnapshot[applied={T|T < 7333976775766757376}]
I20260927 15:07:53.779036 15287 tablet_replica.cc:439] T d7066e40fcd94444ae02b4dfb2876b36 P 36a116f98c2d4cb8b337e3a71a180e8e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } } }
I20260927 15:07:53.779142 15287 tablet_replica.cc:442] T d7066e40fcd94444ae02b4dfb2876b36 P 36a116f98c2d4cb8b337e3a71a180e8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:53.780774 15283 tablet_replica.cc:439] T d7066e40fcd94444ae02b4dfb2876b36 P 5ef67695d9fb4e979f286cb81e737490: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } } }
I20260927 15:07:53.780929 15283 tablet_replica.cc:442] T d7066e40fcd94444ae02b4dfb2876b36 P 5ef67695d9fb4e979f286cb81e737490: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:07:53.786993 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.783463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36012 (local address 127.7.17.194:39663)
0927 15:07:53.783571 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.783574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.783587 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:07:53.783687 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.783689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.783731 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.783793 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:53.784074 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.784652 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.785860 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.786084 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.786174 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.786505 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:53.786590 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:53.786653 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:53.786789 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 15:07:53.786837 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 15:07:53.798243 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.794990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:57589 (local address 127.7.17.194:39663)
0927 15:07:53.795096 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.795101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.795352 (+ 251us) server_negotiation.cc:407] Connection header received
0927 15:07:53.795475 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.795478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.795536 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.795602 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:53.795847 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.796344 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.797244 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.797454 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.797536 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.797796 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:53.797867 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:53.797921 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:53.798007 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 15:07:53.798092 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":35}
I20260927 15:07:53.798254 15328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.794953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:39663 (local address 127.7.17.194:57589)
0927 15:07:53.795270 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:53.795282 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:53.795393 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:53.795607 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:53.795610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:53.795616 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:07:53.795750 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.795755 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.796438 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.796442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.797139 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.797149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.797559 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.797563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.797638 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.797682 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:07:53.797933 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:07:53.797938 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:07:53.797995 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 15:07:53.798123 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":43,"thread_start_us":116,"threads_started":1}
I20260927 15:07:53.804919 15202 raft_consensus.cc:1261] T dccb5cc944c44a33b82ed5242bda8b33 P 5ef67695d9fb4e979f286cb81e737490 [term 1 FOLLOWER]: Refusing update from remote peer 6b1c2b30413046bbb45b6d321ddab1a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:53.805006 14922 raft_consensus.cc:1261] T dccb5cc944c44a33b82ed5242bda8b33 P 36a116f98c2d4cb8b337e3a71a180e8e [term 1 FOLLOWER]: Refusing update from remote peer 6b1c2b30413046bbb45b6d321ddab1a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:53.805305 15307 consensus_queue.cc:1035] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:53.805531 15325 consensus_queue.cc:1035] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:53.842756 14763 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:53.842904 14763 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:53.845067 14763 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9: Sending DeleteTablet for 3 replicas of tablet dccb5cc944c44a33b82ed5242bda8b33
I20260927 15:07:53.845752 14902 tablet_service.cc:1514] Processing DeleteTablet for tablet dccb5cc944c44a33b82ed5242bda8b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:53 UTC) from {username='oryx', principal='oryx/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:54940
I20260927 15:07:53.845752 15182 tablet_service.cc:1514] Processing DeleteTablet for tablet dccb5cc944c44a33b82ed5242bda8b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:53 UTC) from {username='oryx', principal='oryx/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:41444
I20260927 15:07:53.845853 15041 tablet_service.cc:1514] Processing DeleteTablet for tablet dccb5cc944c44a33b82ed5242bda8b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:53 UTC) from {username='oryx', principal='oryx/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:36006
I20260927 15:07:53.845938 15300 tablet_replica.cc:330] T dccb5cc944c44a33b82ed5242bda8b33 P 36a116f98c2d4cb8b337e3a71a180e8e: stopping tablet replica
I20260927 15:07:53.845942 15299 tablet_replica.cc:330] T dccb5cc944c44a33b82ed5242bda8b33 P 5ef67695d9fb4e979f286cb81e737490: stopping tablet replica
I20260927 15:07:53.846067 15299 raft_consensus.cc:2229] T dccb5cc944c44a33b82ed5242bda8b33 P 5ef67695d9fb4e979f286cb81e737490 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:53.846134 15300 raft_consensus.cc:2229] T dccb5cc944c44a33b82ed5242bda8b33 P 36a116f98c2d4cb8b337e3a71a180e8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:53.846185 15299 raft_consensus.cc:2258] T dccb5cc944c44a33b82ed5242bda8b33 P 5ef67695d9fb4e979f286cb81e737490 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:53.846262 15300 raft_consensus.cc:2258] T dccb5cc944c44a33b82ed5242bda8b33 P 36a116f98c2d4cb8b337e3a71a180e8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:53.846349 15301 tablet_replica.cc:330] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2: stopping tablet replica
I20260927 15:07:53.846447 15301 raft_consensus.cc:2229] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:07:53.846725 15301 raft_consensus.cc:2258] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:53.847187 15300 ts_tablet_manager.cc:1905] T dccb5cc944c44a33b82ed5242bda8b33 P 36a116f98c2d4cb8b337e3a71a180e8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:53.847450 15301 ts_tablet_manager.cc:1905] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:53.849613 15301 ts_tablet_manager.cc:1918] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:07:53.849678 15301 log.cc:1196] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-1/wal/wals/dccb5cc944c44a33b82ed5242bda8b33
I20260927 15:07:53.849890 15301 ts_tablet_manager.cc:1939] T dccb5cc944c44a33b82ed5242bda8b33 P 6b1c2b30413046bbb45b6d321ddab1a2: Deleting consensus metadata
I20260927 15:07:53.850039 15300 ts_tablet_manager.cc:1918] T dccb5cc944c44a33b82ed5242bda8b33 P 36a116f98c2d4cb8b337e3a71a180e8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:07:53.850116 15300 log.cc:1196] T dccb5cc944c44a33b82ed5242bda8b33 P 36a116f98c2d4cb8b337e3a71a180e8e: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-0/wal/wals/dccb5cc944c44a33b82ed5242bda8b33
I20260927 15:07:53.850332 15300 ts_tablet_manager.cc:1939] T dccb5cc944c44a33b82ed5242bda8b33 P 36a116f98c2d4cb8b337e3a71a180e8e: Deleting consensus metadata
I20260927 15:07:53.850719 14752 catalog_manager.cc:4841] TS 6b1c2b30413046bbb45b6d321ddab1a2 (127.7.17.194:39663): tablet dccb5cc944c44a33b82ed5242bda8b33 (table test-table [id=c3503f76cc844d8a990fab96b579d793]) successfully deleted
I20260927 15:07:53.850917 14752 catalog_manager.cc:4841] TS 36a116f98c2d4cb8b337e3a71a180e8e (127.7.17.193:41869): tablet dccb5cc944c44a33b82ed5242bda8b33 (table test-table [id=c3503f76cc844d8a990fab96b579d793]) successfully deleted
I20260927 15:07:53.851461 15299 ts_tablet_manager.cc:1905] T dccb5cc944c44a33b82ed5242bda8b33 P 5ef67695d9fb4e979f286cb81e737490: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:53.853873 15299 ts_tablet_manager.cc:1918] T dccb5cc944c44a33b82ed5242bda8b33 P 5ef67695d9fb4e979f286cb81e737490: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:07:53.853940 15299 log.cc:1196] T dccb5cc944c44a33b82ed5242bda8b33 P 5ef67695d9fb4e979f286cb81e737490: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/dccb5cc944c44a33b82ed5242bda8b33
I20260927 15:07:53.854131 15299 ts_tablet_manager.cc:1939] T dccb5cc944c44a33b82ed5242bda8b33 P 5ef67695d9fb4e979f286cb81e737490: Deleting consensus metadata
I20260927 15:07:53.854976 14749 catalog_manager.cc:4841] TS 5ef67695d9fb4e979f286cb81e737490 (127.7.17.195:41521): tablet dccb5cc944c44a33b82ed5242bda8b33 (table test-table [id=c3503f76cc844d8a990fab96b579d793]) successfully deleted
Sep 27 15:07:53 dist-test-slave-hhpq krb5kdc[14720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:07:53 dist-test-slave-hhpq krb5kdc[14720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:53 dist-test-slave-hhpq krb5kdc[14720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 15:07:53.877558 15315 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:53.870362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46658 (local address 127.7.17.254:40831)
0927 15:07:53.870511 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.870516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.870606 (+ 90us) server_negotiation.cc:407] Connection header received
0927 15:07:53.870727 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.870731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.870788 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.870889 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:53.871675 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.872487 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.873477 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.873834 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.873956 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.877406 (+ 3450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46658: BlockingRecv error: recv got EOF from 127.0.0.1:46658 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 15:07:53 dist-test-slave-hhpq krb5kdc[14720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM
I20260927 15:07:53.887773 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.878940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46664 (local address 127.7.17.254:40831)
0927 15:07:53.879059 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.879063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.879270 (+ 207us) server_negotiation.cc:407] Connection header received
0927 15:07:53.879364 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.879368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.879423 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.879512 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:53.880100 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.880995 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.881782 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.882123 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.882230 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.884759 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:53.884763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:53.884766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:53.884797 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:53.886279 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:53.886672 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:53.886675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:53.886677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:53.886761 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:53.886959 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:53.886962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:53.886964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:53.887172 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:53.887289 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:53.887482 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 15:07:53.887606 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 15:07:53.890543 14763 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:07:53.891005 14763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:53.895802 14902 tablet_service.cc:1467] Processing CreateTablet for tablet ef67be9dc4b843c499a464070317cf21 (DEFAULT_TABLE table=test-table [id=de3a7af8488648b1bea9f9b6a5d150fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:53.895887 15041 tablet_service.cc:1467] Processing CreateTablet for tablet ef67be9dc4b843c499a464070317cf21 (DEFAULT_TABLE table=test-table [id=de3a7af8488648b1bea9f9b6a5d150fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:53.896102 14902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef67be9dc4b843c499a464070317cf21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:53.896219 15041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef67be9dc4b843c499a464070317cf21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:53.898521 15277 tablet_bootstrap.cc:492] T ef67be9dc4b843c499a464070317cf21 P 36a116f98c2d4cb8b337e3a71a180e8e: Bootstrap starting.
I20260927 15:07:53.898604 15182 tablet_service.cc:1467] Processing CreateTablet for tablet ef67be9dc4b843c499a464070317cf21 (DEFAULT_TABLE table=test-table [id=de3a7af8488648b1bea9f9b6a5d150fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:53.898859 15182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef67be9dc4b843c499a464070317cf21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:53.899407 15277 tablet_bootstrap.cc:654] T ef67be9dc4b843c499a464070317cf21 P 36a116f98c2d4cb8b337e3a71a180e8e: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:53.900568 15277 tablet_bootstrap.cc:492] T ef67be9dc4b843c499a464070317cf21 P 36a116f98c2d4cb8b337e3a71a180e8e: No bootstrap required, opened a new log
I20260927 15:07:53.900660 15277 ts_tablet_manager.cc:1397] T ef67be9dc4b843c499a464070317cf21 P 36a116f98c2d4cb8b337e3a71a180e8e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 15:07:53.900781 15276 tablet_bootstrap.cc:492] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490: Bootstrap starting.
I20260927 15:07:53.900898 15278 tablet_bootstrap.cc:492] T ef67be9dc4b843c499a464070317cf21 P 6b1c2b30413046bbb45b6d321ddab1a2: Bootstrap starting.
I20260927 15:07:53.901127 15277 raft_consensus.cc:348] T ef67be9dc4b843c499a464070317cf21 P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.901240 15277 raft_consensus.cc:374] T ef67be9dc4b843c499a464070317cf21 P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:53.901268 15277 raft_consensus.cc:729] T ef67be9dc4b843c499a464070317cf21 P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36a116f98c2d4cb8b337e3a71a180e8e, State: Initialized, Role: FOLLOWER
I20260927 15:07:53.901381 15277 consensus_queue.cc:260] T ef67be9dc4b843c499a464070317cf21 P 36a116f98c2d4cb8b337e3a71a180e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.901650 15277 ts_tablet_manager.cc:1428] T ef67be9dc4b843c499a464070317cf21 P 36a116f98c2d4cb8b337e3a71a180e8e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:53.901688 15278 tablet_bootstrap.cc:654] T ef67be9dc4b843c499a464070317cf21 P 6b1c2b30413046bbb45b6d321ddab1a2: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:53.901688 15276 tablet_bootstrap.cc:654] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:53.903949 15278 tablet_bootstrap.cc:492] T ef67be9dc4b843c499a464070317cf21 P 6b1c2b30413046bbb45b6d321ddab1a2: No bootstrap required, opened a new log
I20260927 15:07:53.903949 15276 tablet_bootstrap.cc:492] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490: No bootstrap required, opened a new log
I20260927 15:07:53.904048 15278 ts_tablet_manager.cc:1397] T ef67be9dc4b843c499a464070317cf21 P 6b1c2b30413046bbb45b6d321ddab1a2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 15:07:53.904048 15276 ts_tablet_manager.cc:1397] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 15:07:53.904466 15278 raft_consensus.cc:348] T ef67be9dc4b843c499a464070317cf21 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.904466 15276 raft_consensus.cc:348] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.904589 15278 raft_consensus.cc:374] T ef67be9dc4b843c499a464070317cf21 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:53.904589 15276 raft_consensus.cc:374] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:53.904628 15276 raft_consensus.cc:729] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ef67695d9fb4e979f286cb81e737490, State: Initialized, Role: FOLLOWER
I20260927 15:07:53.904628 15278 raft_consensus.cc:729] T ef67be9dc4b843c499a464070317cf21 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1c2b30413046bbb45b6d321ddab1a2, State: Initialized, Role: FOLLOWER
I20260927 15:07:53.904752 15278 consensus_queue.cc:260] T ef67be9dc4b843c499a464070317cf21 P 6b1c2b30413046bbb45b6d321ddab1a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.904752 15276 consensus_queue.cc:260] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.905042 15278 ts_tablet_manager.cc:1428] T ef67be9dc4b843c499a464070317cf21 P 6b1c2b30413046bbb45b6d321ddab1a2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 15:07:53.905051 15276 ts_tablet_manager.cc:1428] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 15:07:53.910130 15283 raft_consensus.cc:482] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:53.910265 15283 raft_consensus.cc:504] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.911345 15283 leader_election.cc:288] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36a116f98c2d4cb8b337e3a71a180e8e (127.7.17.193:41869), 6b1c2b30413046bbb45b6d321ddab1a2 (127.7.17.194:39663)
I20260927 15:07:53.916026 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.911646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:48325 (local address 127.7.17.193:41869)
0927 15:07:53.911766 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.911770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.911978 (+ 208us) server_negotiation.cc:407] Connection header received
0927 15:07:53.912172 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.912175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.912231 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.912306 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:53.912594 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.913280 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.914706 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.915294 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.915385 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.915455 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:53.915799 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 15:07:53.915888 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:07:53.916481 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.911571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:41869 (local address 127.7.17.195:48325)
0927 15:07:53.911891 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:53.911906 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:53.912058 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:53.912333 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:53.912335 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:53.912354 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:53.912494 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.912501 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.913374 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.913377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.914565 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.914577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.915552 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.915555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.915665 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.915689 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:07:53.916237 (+ 548us) client_negotiation.cc:240] Negotiation successful
0927 15:07:53.916315 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":23,"thread_start_us":91,"threads_started":1}
I20260927 15:07:53.916638 15348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.911913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:39663 (local address 127.7.17.195:35645)
0927 15:07:53.912288 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:53.912301 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:53.912423 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:53.912707 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:53.912709 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:53.912728 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:07:53.912887 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.912894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.914109 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.914112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.915343 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:53.915353 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.916273 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.916276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:53.916370 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.916390 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 15:07:53.916460 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 15:07:53.916523 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":36,"thread_start_us":54,"threads_started":1}
I20260927 15:07:53.916707 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.912035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:35645 (local address 127.7.17.194:39663)
0927 15:07:53.912151 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.912155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.912372 (+ 217us) server_negotiation.cc:407] Connection header received
0927 15:07:53.912544 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.912547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.912613 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.912684 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:07:53.913000 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.913657 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.915470 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.916049 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.916138 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.916184 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:53.916493 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 15:07:53.916579 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:07:53.916922 14921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef67be9dc4b843c499a464070317cf21" candidate_uuid: "5ef67695d9fb4e979f286cb81e737490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" is_pre_election: true
I20260927 15:07:53.917084 14921 raft_consensus.cc:2454] T ef67be9dc4b843c499a464070317cf21 P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ef67695d9fb4e979f286cb81e737490 in term 0.
I20260927 15:07:53.917490 15136 leader_election.cc:302] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36a116f98c2d4cb8b337e3a71a180e8e, 5ef67695d9fb4e979f286cb81e737490; no voters:
I20260927 15:07:53.917812 15283 raft_consensus.cc:2781] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:53.917868 15283 raft_consensus.cc:482] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:53.917901 15283 raft_consensus.cc:3037] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:53.918931 15283 raft_consensus.cc:504] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.919299 15283 leader_election.cc:288] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490 [CANDIDATE]: Term 1 election: Requested vote from peers 36a116f98c2d4cb8b337e3a71a180e8e (127.7.17.193:41869), 6b1c2b30413046bbb45b6d321ddab1a2 (127.7.17.194:39663)
I20260927 15:07:53.919508 15062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef67be9dc4b843c499a464070317cf21" candidate_uuid: "5ef67695d9fb4e979f286cb81e737490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" is_pre_election: true
I20260927 15:07:53.919649 15062 raft_consensus.cc:2454] T ef67be9dc4b843c499a464070317cf21 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ef67695d9fb4e979f286cb81e737490 in term 0.
I20260927 15:07:53.920169 14921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef67be9dc4b843c499a464070317cf21" candidate_uuid: "5ef67695d9fb4e979f286cb81e737490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36a116f98c2d4cb8b337e3a71a180e8e"
I20260927 15:07:53.920310 14921 raft_consensus.cc:3037] T ef67be9dc4b843c499a464070317cf21 P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:53.920405 15062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef67be9dc4b843c499a464070317cf21" candidate_uuid: "5ef67695d9fb4e979f286cb81e737490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1c2b30413046bbb45b6d321ddab1a2"
I20260927 15:07:53.920503 15062 raft_consensus.cc:3037] T ef67be9dc4b843c499a464070317cf21 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:53.921456 15062 raft_consensus.cc:2454] T ef67be9dc4b843c499a464070317cf21 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ef67695d9fb4e979f286cb81e737490 in term 1.
I20260927 15:07:53.921487 14921 raft_consensus.cc:2454] T ef67be9dc4b843c499a464070317cf21 P 36a116f98c2d4cb8b337e3a71a180e8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ef67695d9fb4e979f286cb81e737490 in term 1.
I20260927 15:07:53.921802 15136 leader_election.cc:302] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36a116f98c2d4cb8b337e3a71a180e8e, 5ef67695d9fb4e979f286cb81e737490; no voters:
I20260927 15:07:53.922062 15283 raft_consensus.cc:2781] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:53.922274 15283 raft_consensus.cc:686] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490 [term 1 LEADER]: Becoming Leader. State: Replica: 5ef67695d9fb4e979f286cb81e737490, State: Running, Role: LEADER
I20260927 15:07:53.922542 15283 consensus_queue.cc:237] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.925060 14765 catalog_manager.cc:5495] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ef67695d9fb4e979f286cb81e737490 (127.7.17.195). New cstate: current_term: 1 leader_uuid: "5ef67695d9fb4e979f286cb81e737490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } health_report { overall_health: HEALTHY } } }
I20260927 15:07:53.938879 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:53.934902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.7.17.195:41521)
0927 15:07:53.935041 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:53.935044 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:53.935058 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:07:53.935167 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:53.935169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:53.935224 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:53.935306 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:53.935612 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.936239 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.937482 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:53.937700 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:53.937784 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:53.938094 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:53.938538 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:53.938622 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:53.938692 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 15:07:53.938743 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:07:53.941668 14921 raft_consensus.cc:1261] T ef67be9dc4b843c499a464070317cf21 P 36a116f98c2d4cb8b337e3a71a180e8e [term 1 FOLLOWER]: Refusing update from remote peer 5ef67695d9fb4e979f286cb81e737490: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:53.941686 15062 raft_consensus.cc:1261] T ef67be9dc4b843c499a464070317cf21 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 1 FOLLOWER]: Refusing update from remote peer 5ef67695d9fb4e979f286cb81e737490: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:53.942148 15283 consensus_queue.cc:1035] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:53.942359 15349 consensus_queue.cc:1035] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:53.953999 14763 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:53.954174 14763 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:53.956545 14763 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9: Sending DeleteTablet for 3 replicas of tablet ef67be9dc4b843c499a464070317cf21
I20260927 15:07:53.957199 15182 tablet_service.cc:1514] Processing DeleteTablet for tablet ef67be9dc4b843c499a464070317cf21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:53 UTC) from {username='oryx', principal='oryx/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:41444
I20260927 15:07:53.957391 14902 tablet_service.cc:1514] Processing DeleteTablet for tablet ef67be9dc4b843c499a464070317cf21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:53 UTC) from {username='oryx', principal='oryx/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:54940
I20260927 15:07:53.957571 15299 tablet_replica.cc:330] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490: stopping tablet replica
I20260927 15:07:53.957690 15299 raft_consensus.cc:2229] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:07:53.957973 15299 raft_consensus.cc:2258] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:53.958149 15300 tablet_replica.cc:330] T ef67be9dc4b843c499a464070317cf21 P 36a116f98c2d4cb8b337e3a71a180e8e: stopping tablet replica
I20260927 15:07:53.958267 15300 raft_consensus.cc:2229] T ef67be9dc4b843c499a464070317cf21 P 36a116f98c2d4cb8b337e3a71a180e8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:53.958382 15300 raft_consensus.cc:2258] T ef67be9dc4b843c499a464070317cf21 P 36a116f98c2d4cb8b337e3a71a180e8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:53.958870 15041 tablet_service.cc:1514] Processing DeleteTablet for tablet ef67be9dc4b843c499a464070317cf21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:53 UTC) from {username='oryx', principal='oryx/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:36006
I20260927 15:07:53.959075 15301 tablet_replica.cc:330] T ef67be9dc4b843c499a464070317cf21 P 6b1c2b30413046bbb45b6d321ddab1a2: stopping tablet replica
I20260927 15:07:53.959182 15301 raft_consensus.cc:2229] T ef67be9dc4b843c499a464070317cf21 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:53.959204 15300 ts_tablet_manager.cc:1905] T ef67be9dc4b843c499a464070317cf21 P 36a116f98c2d4cb8b337e3a71a180e8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:53.959296 15301 raft_consensus.cc:2258] T ef67be9dc4b843c499a464070317cf21 P 6b1c2b30413046bbb45b6d321ddab1a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:53.960161 15301 ts_tablet_manager.cc:1905] T ef67be9dc4b843c499a464070317cf21 P 6b1c2b30413046bbb45b6d321ddab1a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:53.961396 15299 ts_tablet_manager.cc:1905] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:53.962687 15301 ts_tablet_manager.cc:1918] T ef67be9dc4b843c499a464070317cf21 P 6b1c2b30413046bbb45b6d321ddab1a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:53.962757 15301 log.cc:1196] T ef67be9dc4b843c499a464070317cf21 P 6b1c2b30413046bbb45b6d321ddab1a2: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-1/wal/wals/ef67be9dc4b843c499a464070317cf21
I20260927 15:07:53.962996 15301 ts_tablet_manager.cc:1939] T ef67be9dc4b843c499a464070317cf21 P 6b1c2b30413046bbb45b6d321ddab1a2: Deleting consensus metadata
I20260927 15:07:53.963692 15299 ts_tablet_manager.cc:1918] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:53.963773 15299 log.cc:1196] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/ef67be9dc4b843c499a464070317cf21
I20260927 15:07:53.963995 15299 ts_tablet_manager.cc:1939] T ef67be9dc4b843c499a464070317cf21 P 5ef67695d9fb4e979f286cb81e737490: Deleting consensus metadata
I20260927 15:07:53.964504 14752 catalog_manager.cc:4841] TS 6b1c2b30413046bbb45b6d321ddab1a2 (127.7.17.194:39663): tablet ef67be9dc4b843c499a464070317cf21 (table test-table [id=de3a7af8488648b1bea9f9b6a5d150fb]) successfully deleted
I20260927 15:07:53.964721 14749 catalog_manager.cc:4841] TS 5ef67695d9fb4e979f286cb81e737490 (127.7.17.195:41521): tablet ef67be9dc4b843c499a464070317cf21 (table test-table [id=de3a7af8488648b1bea9f9b6a5d150fb]) successfully deleted
I20260927 15:07:53.965830 15300 ts_tablet_manager.cc:1918] T ef67be9dc4b843c499a464070317cf21 P 36a116f98c2d4cb8b337e3a71a180e8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:07:53.965911 15300 log.cc:1196] T ef67be9dc4b843c499a464070317cf21 P 36a116f98c2d4cb8b337e3a71a180e8e: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-0/wal/wals/ef67be9dc4b843c499a464070317cf21
I20260927 15:07:53.966137 15300 ts_tablet_manager.cc:1939] T ef67be9dc4b843c499a464070317cf21 P 36a116f98c2d4cb8b337e3a71a180e8e: Deleting consensus metadata
I20260927 15:07:53.966869 14752 catalog_manager.cc:4841] TS 36a116f98c2d4cb8b337e3a71a180e8e (127.7.17.193:41869): tablet ef67be9dc4b843c499a464070317cf21 (table test-table [id=de3a7af8488648b1bea9f9b6a5d150fb]) successfully deleted
I20260927 15:07:53.968073 14763 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:07:53.968520 14763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:07:53.973686 14902 tablet_service.cc:1467] Processing CreateTablet for tablet d7c81550b254461f84e7c14208afc8ff (DEFAULT_TABLE table=test-table [id=a410ed1e60e04ee08d3f30da14cec401]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:53.973680 15041 tablet_service.cc:1467] Processing CreateTablet for tablet d7c81550b254461f84e7c14208afc8ff (DEFAULT_TABLE table=test-table [id=a410ed1e60e04ee08d3f30da14cec401]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:53.973989 14902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c81550b254461f84e7c14208afc8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:53.973990 15041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c81550b254461f84e7c14208afc8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:53.976356 15278 tablet_bootstrap.cc:492] T d7c81550b254461f84e7c14208afc8ff P 6b1c2b30413046bbb45b6d321ddab1a2: Bootstrap starting.
I20260927 15:07:53.976428 15277 tablet_bootstrap.cc:492] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e: Bootstrap starting.
I20260927 15:07:53.977237 15277 tablet_bootstrap.cc:654] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:53.977237 15278 tablet_bootstrap.cc:654] T d7c81550b254461f84e7c14208afc8ff P 6b1c2b30413046bbb45b6d321ddab1a2: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:53.978396 15182 tablet_service.cc:1467] Processing CreateTablet for tablet d7c81550b254461f84e7c14208afc8ff (DEFAULT_TABLE table=test-table [id=a410ed1e60e04ee08d3f30da14cec401]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:07:53.978675 15182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c81550b254461f84e7c14208afc8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:53.981017 15276 tablet_bootstrap.cc:492] T d7c81550b254461f84e7c14208afc8ff P 5ef67695d9fb4e979f286cb81e737490: Bootstrap starting.
I20260927 15:07:53.981921 15276 tablet_bootstrap.cc:654] T d7c81550b254461f84e7c14208afc8ff P 5ef67695d9fb4e979f286cb81e737490: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:53.983059 15276 tablet_bootstrap.cc:492] T d7c81550b254461f84e7c14208afc8ff P 5ef67695d9fb4e979f286cb81e737490: No bootstrap required, opened a new log
I20260927 15:07:53.983141 15276 ts_tablet_manager.cc:1397] T d7c81550b254461f84e7c14208afc8ff P 5ef67695d9fb4e979f286cb81e737490: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 15:07:53.983466 15277 tablet_bootstrap.cc:492] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e: No bootstrap required, opened a new log
I20260927 15:07:53.983556 15277 ts_tablet_manager.cc:1397] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260927 15:07:53.983572 15276 raft_consensus.cc:348] T d7c81550b254461f84e7c14208afc8ff P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.983669 15276 raft_consensus.cc:374] T d7c81550b254461f84e7c14208afc8ff P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:53.983695 15276 raft_consensus.cc:729] T d7c81550b254461f84e7c14208afc8ff P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ef67695d9fb4e979f286cb81e737490, State: Initialized, Role: FOLLOWER
I20260927 15:07:53.983801 15276 consensus_queue.cc:260] T d7c81550b254461f84e7c14208afc8ff P 5ef67695d9fb4e979f286cb81e737490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.983928 15277 raft_consensus.cc:348] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.984030 15277 raft_consensus.cc:374] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:53.984050 15276 ts_tablet_manager.cc:1428] T d7c81550b254461f84e7c14208afc8ff P 5ef67695d9fb4e979f286cb81e737490: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 15:07:53.984058 15277 raft_consensus.cc:729] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36a116f98c2d4cb8b337e3a71a180e8e, State: Initialized, Role: FOLLOWER
I20260927 15:07:53.984166 15277 consensus_queue.cc:260] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.984403 15277 ts_tablet_manager.cc:1428] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:07:53.986734 15278 tablet_bootstrap.cc:492] T d7c81550b254461f84e7c14208afc8ff P 6b1c2b30413046bbb45b6d321ddab1a2: No bootstrap required, opened a new log
I20260927 15:07:53.986829 15278 ts_tablet_manager.cc:1397] T d7c81550b254461f84e7c14208afc8ff P 6b1c2b30413046bbb45b6d321ddab1a2: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.001s
I20260927 15:07:53.987211 15278 raft_consensus.cc:348] T d7c81550b254461f84e7c14208afc8ff P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.987326 15278 raft_consensus.cc:374] T d7c81550b254461f84e7c14208afc8ff P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:53.987355 15278 raft_consensus.cc:729] T d7c81550b254461f84e7c14208afc8ff P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1c2b30413046bbb45b6d321ddab1a2, State: Initialized, Role: FOLLOWER
I20260927 15:07:53.987463 15278 consensus_queue.cc:260] T d7c81550b254461f84e7c14208afc8ff P 6b1c2b30413046bbb45b6d321ddab1a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.987720 15278 ts_tablet_manager.cc:1428] T d7c81550b254461f84e7c14208afc8ff P 6b1c2b30413046bbb45b6d321ddab1a2: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260927 15:07:53.992208 15287 raft_consensus.cc:482] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:07:53.992316 15287 raft_consensus.cc:504] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.992705 15287 leader_election.cc:288] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b1c2b30413046bbb45b6d321ddab1a2 (127.7.17.194:39663), 5ef67695d9fb4e979f286cb81e737490 (127.7.17.195:41521)
I20260927 15:07:53.993099 15062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c81550b254461f84e7c14208afc8ff" candidate_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" is_pre_election: true
I20260927 15:07:53.993242 15062 raft_consensus.cc:2454] T d7c81550b254461f84e7c14208afc8ff P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36a116f98c2d4cb8b337e3a71a180e8e in term 0.
I20260927 15:07:53.993321 15202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c81550b254461f84e7c14208afc8ff" candidate_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ef67695d9fb4e979f286cb81e737490" is_pre_election: true
I20260927 15:07:53.993443 15202 raft_consensus.cc:2454] T d7c81550b254461f84e7c14208afc8ff P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36a116f98c2d4cb8b337e3a71a180e8e in term 0.
I20260927 15:07:53.993535 14856 leader_election.cc:302] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36a116f98c2d4cb8b337e3a71a180e8e, 6b1c2b30413046bbb45b6d321ddab1a2; no voters:
I20260927 15:07:53.993831 15287 raft_consensus.cc:2781] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:07:53.993894 15287 raft_consensus.cc:482] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:07:53.993935 15287 raft_consensus.cc:3037] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:53.995167 15287 raft_consensus.cc:504] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.995543 15287 leader_election.cc:288] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e [CANDIDATE]: Term 1 election: Requested vote from peers 6b1c2b30413046bbb45b6d321ddab1a2 (127.7.17.194:39663), 5ef67695d9fb4e979f286cb81e737490 (127.7.17.195:41521)
I20260927 15:07:53.995867 15062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c81550b254461f84e7c14208afc8ff" candidate_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1c2b30413046bbb45b6d321ddab1a2"
I20260927 15:07:53.995919 15202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c81550b254461f84e7c14208afc8ff" candidate_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ef67695d9fb4e979f286cb81e737490"
I20260927 15:07:53.995982 15062 raft_consensus.cc:3037] T d7c81550b254461f84e7c14208afc8ff P 6b1c2b30413046bbb45b6d321ddab1a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:53.996026 15202 raft_consensus.cc:3037] T d7c81550b254461f84e7c14208afc8ff P 5ef67695d9fb4e979f286cb81e737490 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:53.997138 15062 raft_consensus.cc:2454] T d7c81550b254461f84e7c14208afc8ff P 6b1c2b30413046bbb45b6d321ddab1a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36a116f98c2d4cb8b337e3a71a180e8e in term 1.
I20260927 15:07:53.997594 14856 leader_election.cc:302] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36a116f98c2d4cb8b337e3a71a180e8e, 6b1c2b30413046bbb45b6d321ddab1a2; no voters:
I20260927 15:07:53.997689 15202 raft_consensus.cc:2454] T d7c81550b254461f84e7c14208afc8ff P 5ef67695d9fb4e979f286cb81e737490 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36a116f98c2d4cb8b337e3a71a180e8e in term 1.
I20260927 15:07:53.997815 15287 raft_consensus.cc:2781] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:53.997953 15287 raft_consensus.cc:686] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e [term 1 LEADER]: Becoming Leader. State: Replica: 36a116f98c2d4cb8b337e3a71a180e8e, State: Running, Role: LEADER
I20260927 15:07:53.998101 15287 consensus_queue.cc:237] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } }
I20260927 15:07:53.999925 14765 catalog_manager.cc:5495] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 36a116f98c2d4cb8b337e3a71a180e8e (127.7.17.193). New cstate: current_term: 1 leader_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a116f98c2d4cb8b337e3a71a180e8e" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 } health_report { overall_health: UNKNOWN } } }
I20260927 15:07:54.018568 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:54.014284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.7.17.193:41869)
0927 15:07:54.014384 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:54.014387 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:54.014398 (+ 11us) server_negotiation.cc:407] Connection header received
0927 15:07:54.014436 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:54.014439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:54.014481 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:54.014550 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:54.014889 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:54.015622 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:54.016919 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:54.017264 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:54.017360 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:54.017775 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:54.017862 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:54.017923 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:54.018362 (+ 439us) server_negotiation.cc:299] Negotiation successful
0927 15:07:54.018423 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:07:54.029925 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:54.026073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:48017 (local address 127.7.17.194:39663)
0927 15:07:54.026187 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:54.026191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:54.026205 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:07:54.026306 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:54.026310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:54.026360 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:54.026432 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:54.026672 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:54.027397 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:54.028643 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:54.029028 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:54.029137 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:54.029445 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:54.029548 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:54.029620 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:54.029718 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 15:07:54.029780 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:07:54.029948 15363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:54.025771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:39663 (local address 127.7.17.193:48017)
0927 15:07:54.026090 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:54.026105 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:54.026205 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:54.026455 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:54.026458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:54.026465 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:07:54.026576 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:54.026582 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:54.027512 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:54.027515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:54.028507 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:54.028519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:54.029142 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:54.029146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:54.029238 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:54.029299 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:07:54.029629 (+ 330us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:07:54.029635 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:07:54.029696 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 15:07:54.029787 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":107,"threads_started":1}
I20260927 15:07:54.038166 15062 raft_consensus.cc:1261] T d7c81550b254461f84e7c14208afc8ff P 6b1c2b30413046bbb45b6d321ddab1a2 [term 1 FOLLOWER]: Refusing update from remote peer 36a116f98c2d4cb8b337e3a71a180e8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:54.038604 15287 consensus_queue.cc:1035] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b1c2b30413046bbb45b6d321ddab1a2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:54.038703 15202 raft_consensus.cc:1261] T d7c81550b254461f84e7c14208afc8ff P 5ef67695d9fb4e979f286cb81e737490 [term 1 FOLLOWER]: Refusing update from remote peer 36a116f98c2d4cb8b337e3a71a180e8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:07:54.039111 15287 consensus_queue.cc:1035] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ef67695d9fb4e979f286cb81e737490" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:07:54.065671 15328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:54.061587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:41869 (local address 127.7.17.194:42895)
0927 15:07:54.061706 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:54.061721 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:54.061821 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:54.062098 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:54.062101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:54.062108 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:07:54.062204 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:54.062210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:54.063110 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:54.063113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:54.064072 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:54.064084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:54.064920 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:54.064924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:54.065016 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:54.065049 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:07:54.065366 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:07:54.065372 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:07:54.065421 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 15:07:54.065508 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 15:07:54.066087 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:54.061800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:42895 (local address 127.7.17.193:41869)
0927 15:07:54.061905 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:54.061909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:54.061921 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:07:54.061960 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:54.061963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:54.062022 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:54.062089 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:07:54.062321 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:54.062981 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:54.064213 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:54.064680 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:54.064850 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:54.065167 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:07:54.065268 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:07:54.065318 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:54.065914 (+ 596us) server_negotiation.cc:299] Negotiation successful
0927 15:07:54.065969 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 15:07:54.095872 14765 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:54.096055 14765 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:07:54.098346 14765 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd7c17b011a8423a849d956de0960bc9: Sending DeleteTablet for 3 replicas of tablet d7c81550b254461f84e7c14208afc8ff
I20260927 15:07:54.098989 15182 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c81550b254461f84e7c14208afc8ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:54 UTC) from {username='oryx', principal='oryx/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:41444
I20260927 15:07:54.099032 14902 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c81550b254461f84e7c14208afc8ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:54 UTC) from {username='oryx', principal='oryx/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:54940
I20260927 15:07:54.099181 15299 tablet_replica.cc:330] T d7c81550b254461f84e7c14208afc8ff P 5ef67695d9fb4e979f286cb81e737490: stopping tablet replica
I20260927 15:07:54.099285 15299 raft_consensus.cc:2229] T d7c81550b254461f84e7c14208afc8ff P 5ef67695d9fb4e979f286cb81e737490 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:54.099313 15300 tablet_replica.cc:330] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e: stopping tablet replica
I20260927 15:07:54.099407 15299 raft_consensus.cc:2258] T d7c81550b254461f84e7c14208afc8ff P 5ef67695d9fb4e979f286cb81e737490 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:54.099429 15300 raft_consensus.cc:2229] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:07:54.099650 15300 raft_consensus.cc:2258] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:54.099799 15041 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c81550b254461f84e7c14208afc8ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:07:54 UTC) from {username='oryx', principal='oryx/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:36006
I20260927 15:07:54.099964 15301 tablet_replica.cc:330] T d7c81550b254461f84e7c14208afc8ff P 6b1c2b30413046bbb45b6d321ddab1a2: stopping tablet replica
I20260927 15:07:54.100076 15301 raft_consensus.cc:2229] T d7c81550b254461f84e7c14208afc8ff P 6b1c2b30413046bbb45b6d321ddab1a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:07:54.100186 15301 raft_consensus.cc:2258] T d7c81550b254461f84e7c14208afc8ff P 6b1c2b30413046bbb45b6d321ddab1a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:07:54.100489 15300 ts_tablet_manager.cc:1905] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:54.100874 15301 ts_tablet_manager.cc:1905] T d7c81550b254461f84e7c14208afc8ff P 6b1c2b30413046bbb45b6d321ddab1a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:54.102650 15300 ts_tablet_manager.cc:1918] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:07:54.102723 15300 log.cc:1196] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-0/wal/wals/d7c81550b254461f84e7c14208afc8ff
I20260927 15:07:54.102928 15300 ts_tablet_manager.cc:1939] T d7c81550b254461f84e7c14208afc8ff P 36a116f98c2d4cb8b337e3a71a180e8e: Deleting consensus metadata
I20260927 15:07:54.103410 15301 ts_tablet_manager.cc:1918] T d7c81550b254461f84e7c14208afc8ff P 6b1c2b30413046bbb45b6d321ddab1a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:07:54.103492 15301 log.cc:1196] T d7c81550b254461f84e7c14208afc8ff P 6b1c2b30413046bbb45b6d321ddab1a2: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-1/wal/wals/d7c81550b254461f84e7c14208afc8ff
I20260927 15:07:54.103641 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 14837
I20260927 15:07:54.103722 15301 ts_tablet_manager.cc:1939] T d7c81550b254461f84e7c14208afc8ff P 6b1c2b30413046bbb45b6d321ddab1a2: Deleting consensus metadata
I20260927 15:07:54.104645 14752 catalog_manager.cc:4841] TS 36a116f98c2d4cb8b337e3a71a180e8e (127.7.17.193:41869): tablet d7c81550b254461f84e7c14208afc8ff (table test-table [id=a410ed1e60e04ee08d3f30da14cec401]) successfully deleted
I20260927 15:07:54.105046 14752 catalog_manager.cc:4841] TS 6b1c2b30413046bbb45b6d321ddab1a2 (127.7.17.194:39663): tablet d7c81550b254461f84e7c14208afc8ff (table test-table [id=a410ed1e60e04ee08d3f30da14cec401]) successfully deleted
I20260927 15:07:54.109581 15299 ts_tablet_manager.cc:1905] T d7c81550b254461f84e7c14208afc8ff P 5ef67695d9fb4e979f286cb81e737490: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:07:54.112318 15299 ts_tablet_manager.cc:1918] T d7c81550b254461f84e7c14208afc8ff P 5ef67695d9fb4e979f286cb81e737490: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:07:54.112423 15299 log.cc:1196] T d7c81550b254461f84e7c14208afc8ff P 5ef67695d9fb4e979f286cb81e737490: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/d7c81550b254461f84e7c14208afc8ff
I20260927 15:07:54.112807 15299 ts_tablet_manager.cc:1939] T d7c81550b254461f84e7c14208afc8ff P 5ef67695d9fb4e979f286cb81e737490: Deleting consensus metadata
I20260927 15:07:54.113093 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 14977
I20260927 15:07:54.114586 14749 catalog_manager.cc:4841] TS 5ef67695d9fb4e979f286cb81e737490 (127.7.17.195:41521): tablet d7c81550b254461f84e7c14208afc8ff (table test-table [id=a410ed1e60e04ee08d3f30da14cec401]) successfully deleted
I20260927 15:07:54.119763 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 15117
I20260927 15:07:54.125257 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 14733
2026-09-27T15:07:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3140 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:07:54 dist-test-slave-hhpq krb5kdc[15370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:07:54 dist-test-slave-hhpq krb5kdc[15370](info): set up 2 sockets
Sep 27 15:07:54 dist-test-slave-hhpq krb5kdc[15370](info): commencing operation
krb5kdc: starting...
W20260927 15:07:55.081475 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:07:55 dist-test-slave-hhpq krb5kdc[15370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:07:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:07:55.204068 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:36569
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:34095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:36569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:55.316687 15383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:55.317018 15383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:55.317080 15383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:55.321234 15383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:55.321311 15383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:55.321337 15383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:55.321358 15383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:55.321377 15383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:55.326933 15383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:34095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:36569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:36569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:55.328125 15383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:55.329236 15383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:55.337661 15388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:55.337656 15391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:55.337653 15389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:55.337920 15383 server_base.cc:1034] running on GCE node
I20260927 15:07:55.338583 15383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:55.339763 15383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:55.340906 15383 hybrid_clock.cc:648] HybridClock initialized: now 1790521675340887 us; error 40 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T15:07:55Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1227 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:07:55 dist-test-slave-hhpq krb5kdc[15396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:07:55 dist-test-slave-hhpq krb5kdc[15396](info): set up 2 sockets
Sep 27 15:07:55 dist-test-slave-hhpq krb5kdc[15396](info): commencing operation
krb5kdc: starting...
W20260927 15:07:56.309520 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:07:56 dist-test-slave-hhpq krb5kdc[15396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:07:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:07:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:07:56.431658 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:46207
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:46207
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:56.541603 15409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:56.541898 15409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:56.541940 15409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:56.546028 15409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:56.546106 15409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:56.546124 15409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:56.546141 15409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:56.546154 15409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:56.551724 15409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:46207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:46207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:56.552942 15409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:56.554010 15409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:56.562891 15417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:56.562919 15414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:56.562908 15415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:56.563231 15409 server_base.cc:1034] running on GCE node
I20260927 15:07:56.563941 15409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:56.565173 15409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:56.566349 15409 hybrid_clock.cc:648] HybridClock initialized: now 1790521676566323 us; error 41 us; skew 500 ppm
Sep 27 15:07:56 dist-test-slave-hhpq krb5kdc[15396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521676, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:56.569437 15409 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:07:56.570480 15409 webserver.cc:460] Webserver started at http://127.7.17.254:33385/ using document root <none> and password file <none>
I20260927 15:07:56.571058 15409 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:56.571135 15409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:56.571382 15409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:56.573536 15409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "6eb96c7b7a1241198fc952e9f163e983"
format_stamp: "Formatted at 2026-09-27 15:07:56 on dist-test-slave-hhpq"
server_key: "12f66a00bbcc5d322f0fccb99e2f6585"
server_key_iv: "2524f5a2876ebcfdbe381ab77c9b8cf1"
server_key_version: "encryptionkey@0"
I20260927 15:07:56.574116 15409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "6eb96c7b7a1241198fc952e9f163e983"
format_stamp: "Formatted at 2026-09-27 15:07:56 on dist-test-slave-hhpq"
server_key: "12f66a00bbcc5d322f0fccb99e2f6585"
server_key_iv: "2524f5a2876ebcfdbe381ab77c9b8cf1"
server_key_version: "encryptionkey@0"
I20260927 15:07:56.577940 15409 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:07:56.580410 15424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:56.581743 15409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:07:56.581866 15409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "6eb96c7b7a1241198fc952e9f163e983"
format_stamp: "Formatted at 2026-09-27 15:07:56 on dist-test-slave-hhpq"
server_key: "12f66a00bbcc5d322f0fccb99e2f6585"
server_key_iv: "2524f5a2876ebcfdbe381ab77c9b8cf1"
server_key_version: "encryptionkey@0"
I20260927 15:07:56.581969 15409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:56.592702 15409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:56.595671 15409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:56.595854 15409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:56.602727 15409 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:46207
I20260927 15:07:56.602753 15476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:46207 every 8 connection(s)
I20260927 15:07:56.603724 15409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:56.606541 15477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:07:56.608181 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 15409
I20260927 15:07:56.608304 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:07:56.608596 7239 external_mini_cluster.cc:1428] Setting key 38dc402a91e677180525e693b4054faf
I20260927 15:07:56.612723 15477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983: Bootstrap starting.
I20260927 15:07:56.614945 15477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983: Neither blocks nor log segments found. Creating new log.
I20260927 15:07:56.615690 15477 log.cc:824] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:07:56 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:56.617753 15477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983: No bootstrap required, opened a new log
I20260927 15:07:56.621256 15477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb96c7b7a1241198fc952e9f163e983" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46207 } }
I20260927 15:07:56.621531 15477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:07:56.621592 15477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eb96c7b7a1241198fc952e9f163e983, State: Initialized, Role: FOLLOWER
I20260927 15:07:56.622040 15477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb96c7b7a1241198fc952e9f163e983" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46207 } }
I20260927 15:07:56.622177 15477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:56.622237 15477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:56.622327 15477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:07:56.622396 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:56.610125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58536 (local address 127.7.17.254:46207)
0927 15:07:56.610554 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:56.610569 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:56.610606 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:07:56.611249 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:56.611266 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:56.611599 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:56.611904 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:56.612683 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:56.613725 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:56.614585 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:56.615099 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:56.615240 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:56.617853 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:56.617860 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:56.617877 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:56.617909 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:56.619552 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:56.619962 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:56.619967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:56.619973 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:56.620118 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:56.620372 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:56.620376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:56.620377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:56.620764 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:56.620997 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:56.621248 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 15:07:56.621556 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":115,"threads_started":1}
I20260927 15:07:56.623219 15477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb96c7b7a1241198fc952e9f163e983" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46207 } }
I20260927 15:07:56.623565 15477 leader_election.cc:302] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6eb96c7b7a1241198fc952e9f163e983; no voters:
I20260927 15:07:56.623896 15477 leader_election.cc:288] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:07:56.624051 15482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:07:56.624240 15482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [term 1 LEADER]: Becoming Leader. State: Replica: 6eb96c7b7a1241198fc952e9f163e983, State: Running, Role: LEADER
I20260927 15:07:56.624570 15482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb96c7b7a1241198fc952e9f163e983" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46207 } }
I20260927 15:07:56.625108 15477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:56.626266 15483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6eb96c7b7a1241198fc952e9f163e983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb96c7b7a1241198fc952e9f163e983" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46207 } } }
I20260927 15:07:56.626243 15484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6eb96c7b7a1241198fc952e9f163e983. Latest consensus state: current_term: 1 leader_uuid: "6eb96c7b7a1241198fc952e9f163e983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb96c7b7a1241198fc952e9f163e983" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46207 } } }
I20260927 15:07:56.626367 15483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:56.626367 15484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:56.626753 15491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:56.629418 15491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:56.635409 15491 catalog_manager.cc:1347] Generated new cluster ID: 91d41833e8ae4450bc8fdb1f391cc2d1
I20260927 15:07:56.635488 15491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:56.645020 15491 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:07:56.646353 15491 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:56.666301 15491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983: Generated new TSK 0
I20260927 15:07:56.667223 15491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
I20260927 15:07:56.700161 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46207
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:56.812618 15503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:56.812927 15503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:56.812971 15503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:56.817190 15503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:56.817262 15503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:56.817325 15503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:07:56.822815 15503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:56.823949 15503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:56.825012 15503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:56.834895 15511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:56.834895 15508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:56.834982 15503 server_base.cc:1034] running on GCE node
W20260927 15:07:56.834924 15509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:56.835609 15503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:56.836319 15503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:56.837493 15503 hybrid_clock.cc:648] HybridClock initialized: now 1790521676837468 us; error 39 us; skew 500 ppm
Sep 27 15:07:56 dist-test-slave-hhpq krb5kdc[15396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521676, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:56.840507 15503 init.cc:377] Logged in from keytab as kudu/127.7.17.193@KRBTEST.COM (short username kudu)
I20260927 15:07:56.841611 15503 webserver.cc:460] Webserver started at http://127.7.17.193:33809/ using document root <none> and password file <none>
I20260927 15:07:56.842193 15503 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:56.842262 15503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:56.842463 15503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:56.844587 15503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "496d63d5525e43e4a6a548e8dcf4cba8"
format_stamp: "Formatted at 2026-09-27 15:07:56 on dist-test-slave-hhpq"
server_key: "02058f8692ad23da35ed65a1ebaf2a1d"
server_key_iv: "83f720547c79fce62d5a2cf5614204f3"
server_key_version: "encryptionkey@0"
I20260927 15:07:56.845183 15503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "496d63d5525e43e4a6a548e8dcf4cba8"
format_stamp: "Formatted at 2026-09-27 15:07:56 on dist-test-slave-hhpq"
server_key: "02058f8692ad23da35ed65a1ebaf2a1d"
server_key_iv: "83f720547c79fce62d5a2cf5614204f3"
server_key_version: "encryptionkey@0"
I20260927 15:07:56.849009 15503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:07:56.851645 15518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:56.853248 15503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:56.853397 15503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "496d63d5525e43e4a6a548e8dcf4cba8"
format_stamp: "Formatted at 2026-09-27 15:07:56 on dist-test-slave-hhpq"
server_key: "02058f8692ad23da35ed65a1ebaf2a1d"
server_key_iv: "83f720547c79fce62d5a2cf5614204f3"
server_key_version: "encryptionkey@0"
I20260927 15:07:56.853513 15503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:56.879896 15503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:56.883558 15503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:56.883760 15503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:56.884392 15503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:56.885353 15503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:56.885404 15503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:56.885459 15503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:56.885488 15503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:56.894466 15503 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:42371
I20260927 15:07:56.894493 15631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:42371 every 8 connection(s)
I20260927 15:07:56.895413 15503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:56.896592 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 15503
I20260927 15:07:56.896715 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:07:56.897047 7239 external_mini_cluster.cc:1428] Setting key 282fa5acb88709f01fc74f8bc1850037
WARNING: no policy specified for kudu/127.7.17.194@KRBTEST.COM; defaulting to no policy
Sep 27 15:07:56 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521676, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.194@KRBTEST.COM" created.
I20260927 15:07:56.910460 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:56.897518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:35899 (local address 127.7.17.254:46207)
0927 15:07:56.897673 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:56.897678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:56.898926 (+ 1248us) server_negotiation.cc:407] Connection header received
0927 15:07:56.899835 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:56.899838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:56.899893 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:56.899968 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:56.901308 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:56.901840 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:56.903045 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:56.903274 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:56.903363 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:56.906186 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:56.906189 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:56.906191 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:56.906220 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:56.907513 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:56.908343 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:56.908347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:56.908349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:56.908450 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:56.909059 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:56.909062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:56.909063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:56.909295 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:56.909378 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:56.910137 (+ 759us) server_negotiation.cc:299] Negotiation successful
0927 15:07:56.910286 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:07:56.911283 15634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:56.897986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.193:35899)
0927 15:07:56.898742 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:56.898800 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:56.899613 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:56.900183 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:56.900191 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:56.900574 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:56.901143 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:56.901159 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:56.901971 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:56.901973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:56.902909 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:56.902919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:56.903431 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:56.903435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:56.903613 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:56.904205 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:56.904224 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:56.905966 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:56.907675 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:56.907681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:56.907696 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:56.908205 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:56.908599 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:56.908603 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:56.908605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:56.908857 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:56.909398 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:56.909405 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:56.909793 (+ 388us) client_negotiation.cc:769] Sending connection context
0927 15:07:56.910043 (+ 250us) client_negotiation.cc:240] Negotiation successful
0927 15:07:56.910348 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":120,"threads_started":1}
I20260927 15:07:56.912608 15632 heartbeater.cc:347] Connected to a master server at 127.7.17.254:46207
I20260927 15:07:56.912882 15632 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:56.913417 15632 heartbeater.cc:511] Master 127.7.17.254:46207 requested a full tablet report, sending...
I20260927 15:07:56.915194 15441 ts_manager.cc:194] Registered new tserver with Master: 496d63d5525e43e4a6a548e8dcf4cba8 (127.7.17.193:42371)
I20260927 15:07:56.916815 15441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:35899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
I20260927 15:07:56.928201 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46207
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:57.041198 15637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:57.041487 15637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:57.041532 15637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:57.045814 15637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:57.045887 15637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:57.045953 15637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:07:57.051406 15637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:57.052457 15637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:57.053534 15637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:57.063489 15642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:57.063505 15645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:57.063580 15637 server_base.cc:1034] running on GCE node
W20260927 15:07:57.063480 15643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:57.064215 15637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:57.064940 15637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:57.066121 15637 hybrid_clock.cc:648] HybridClock initialized: now 1790521677066086 us; error 52 us; skew 500 ppm
Sep 27 15:07:57 dist-test-slave-hhpq krb5kdc[15396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521677, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:57.069139 15637 init.cc:377] Logged in from keytab as kudu/127.7.17.194@KRBTEST.COM (short username kudu)
I20260927 15:07:57.070273 15637 webserver.cc:460] Webserver started at http://127.7.17.194:39043/ using document root <none> and password file <none>
I20260927 15:07:57.071079 15637 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:57.071153 15637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:57.071333 15637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:57.073527 15637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "1f88d69e05e146bb80e3076425201a98"
format_stamp: "Formatted at 2026-09-27 15:07:57 on dist-test-slave-hhpq"
server_key: "b543218da1fddab7b5aae90db448a832"
server_key_iv: "d4bbb7fa00b729ae6f0ce49ffaa7b0d3"
server_key_version: "encryptionkey@0"
I20260927 15:07:57.074107 15637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "1f88d69e05e146bb80e3076425201a98"
format_stamp: "Formatted at 2026-09-27 15:07:57 on dist-test-slave-hhpq"
server_key: "b543218da1fddab7b5aae90db448a832"
server_key_iv: "d4bbb7fa00b729ae6f0ce49ffaa7b0d3"
server_key_version: "encryptionkey@0"
I20260927 15:07:57.077952 15637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:07:57.080492 15652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:57.081843 15637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:57.081952 15637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "1f88d69e05e146bb80e3076425201a98"
format_stamp: "Formatted at 2026-09-27 15:07:57 on dist-test-slave-hhpq"
server_key: "b543218da1fddab7b5aae90db448a832"
server_key_iv: "d4bbb7fa00b729ae6f0ce49ffaa7b0d3"
server_key_version: "encryptionkey@0"
I20260927 15:07:57.082048 15637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:57.103718 15637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:57.107185 15637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:57.107355 15637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:57.107945 15637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:57.108783 15637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:57.108861 15637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:57.108938 15637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:57.108971 15637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:57.117509 15637 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:44801
I20260927 15:07:57.117537 15765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:44801 every 8 connection(s)
I20260927 15:07:57.118405 15637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:07:57.124935 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 15637
I20260927 15:07:57.125072 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:07:57.125329 7239 external_mini_cluster.cc:1428] Setting key 9f690ba78bd7f09d9f80c3279e628218
Sep 27 15:07:57 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521677, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:57.132818 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.120223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:57035 (local address 127.7.17.254:46207)
0927 15:07:57.120363 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:57.120367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:57.121272 (+ 905us) server_negotiation.cc:407] Connection header received
0927 15:07:57.122690 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:57.122694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:57.122743 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:57.122819 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:57.124065 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.124598 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.126216 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.126543 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.126648 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.129255 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:57.129258 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:57.129260 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:57.129282 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:57.130777 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:57.131291 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:57.131294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:57.131296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:57.131375 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:57.131701 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:57.131704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:57.131705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:57.131892 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:57.131976 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:57.132517 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 15:07:57.132648 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.7.17.195@KRBTEST.COM; defaulting to no policy
I20260927 15:07:57.133716 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.120655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.194:57035)
0927 15:07:57.121121 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:57.121167 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:57.122472 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:57.123025 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:57.123032 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:57.123382 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:57.123869 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.123882 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.124768 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.124772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.126055 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.126067 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.126759 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.126761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.126927 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.127525 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:57.127542 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:57.129091 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:57.130893 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:57.130900 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:57.130918 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:57.131183 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:57.131455 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:57.131458 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:57.131460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:57.131604 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:57.131998 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:57.132006 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:57.132256 (+ 250us) client_negotiation.cc:769] Sending connection context
0927 15:07:57.132430 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 15:07:57.132779 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":124,"threads_started":1}
I20260927 15:07:57.135018 15766 heartbeater.cc:347] Connected to a master server at 127.7.17.254:46207
I20260927 15:07:57.135267 15766 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:57.135784 15766 heartbeater.cc:511] Master 127.7.17.254:46207 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.195@KRBTEST.COM" created.
I20260927 15:07:57.137036 15441 ts_manager.cc:194] Registered new tserver with Master: 1f88d69e05e146bb80e3076425201a98 (127.7.17.194:44801)
I20260927 15:07:57.137619 15441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:57035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
I20260927 15:07:57.152472 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46207
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:57.262357 15771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:57.262672 15771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:57.262718 15771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:57.266927 15771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:57.267020 15771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:57.267083 15771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:07:57.272681 15771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:57.273803 15771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:57.274906 15771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:57.285202 15777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:57.285218 15776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:57.285224 15779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:57.285547 15771 server_base.cc:1034] running on GCE node
I20260927 15:07:57.286006 15771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:57.286729 15771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:57.287917 15771 hybrid_clock.cc:648] HybridClock initialized: now 1790521677287881 us; error 50 us; skew 500 ppm
Sep 27 15:07:57 dist-test-slave-hhpq krb5kdc[15396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521677, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:57.290853 15771 init.cc:377] Logged in from keytab as kudu/127.7.17.195@KRBTEST.COM (short username kudu)
I20260927 15:07:57.291900 15771 webserver.cc:460] Webserver started at http://127.7.17.195:44833/ using document root <none> and password file <none>
I20260927 15:07:57.292478 15771 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:57.292547 15771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:57.292744 15771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:07:57.294879 15771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "2ee353359b4f45019eb51f8d75497748"
format_stamp: "Formatted at 2026-09-27 15:07:57 on dist-test-slave-hhpq"
server_key: "611b2dfe2a29b4c5c52e7df4862ab4a6"
server_key_iv: "2296917883b3d3480e1c3ba7aa4022f6"
server_key_version: "encryptionkey@0"
I20260927 15:07:57.295451 15771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "2ee353359b4f45019eb51f8d75497748"
format_stamp: "Formatted at 2026-09-27 15:07:57 on dist-test-slave-hhpq"
server_key: "611b2dfe2a29b4c5c52e7df4862ab4a6"
server_key_iv: "2296917883b3d3480e1c3ba7aa4022f6"
server_key_version: "encryptionkey@0"
I20260927 15:07:57.299494 15771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:07:57.302350 15786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:57.303704 15771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:07:57.303835 15771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "2ee353359b4f45019eb51f8d75497748"
format_stamp: "Formatted at 2026-09-27 15:07:57 on dist-test-slave-hhpq"
server_key: "611b2dfe2a29b4c5c52e7df4862ab4a6"
server_key_iv: "2296917883b3d3480e1c3ba7aa4022f6"
server_key_version: "encryptionkey@0"
I20260927 15:07:57.303944 15771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:57.323608 15771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:57.327095 15771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:57.327265 15771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:57.327884 15771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:57.328747 15771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:57.328804 15771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:57.328874 15771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:57.328896 15771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:57.337272 15771 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:43415
I20260927 15:07:57.337298 15899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:43415 every 8 connection(s)
I20260927 15:07:57.338204 15771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:57.339160 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 15771
I20260927 15:07:57.339277 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:07:57.339525 7239 external_mini_cluster.cc:1428] Setting key 4b3107d400039eefef0457deac009e8c
Sep 27 15:07:57 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521677, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:07:57.352505 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.340205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:45121 (local address 127.7.17.254:46207)
0927 15:07:57.340350 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:57.340354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:57.341288 (+ 934us) server_negotiation.cc:407] Connection header received
0927 15:07:57.342633 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:57.342636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:57.342683 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:57.342756 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:57.344015 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.344567 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.345857 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.346137 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.346222 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.348887 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:57.348890 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:57.348892 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:57.348984 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:57.350330 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:57.350830 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:57.350832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:57.350834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:57.350973 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:57.351374 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:57.351377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:57.351378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:57.351578 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:57.351672 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:57.352235 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 15:07:57.352368 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:07:57.353605 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.340571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.195:45121)
0927 15:07:57.341116 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:57.341185 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:57.342391 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:57.342966 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:57.342973 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:57.343339 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:57.343870 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.343884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.344706 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.344709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.345731 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.345743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.346277 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.346279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.346466 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.347114 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:57.347132 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:57.348680 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:07:57.350446 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:57.350452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:57.350467 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:57.350744 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:57.351092 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:07:57.351095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:07:57.351097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:07:57.351267 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:07:57.351680 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:07:57.351686 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:07:57.351948 (+ 262us) client_negotiation.cc:769] Sending connection context
0927 15:07:57.352175 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 15:07:57.352526 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":153,"threads_started":1}
I20260927 15:07:57.354730 15900 heartbeater.cc:347] Connected to a master server at 127.7.17.254:46207
I20260927 15:07:57.354978 15900 heartbeater.cc:464] Registering TS with master...
I20260927 15:07:57.355466 15900 heartbeater.cc:511] Master 127.7.17.254:46207 requested a full tablet report, sending...
I20260927 15:07:57.356495 15441 ts_manager.cc:194] Registered new tserver with Master: 2ee353359b4f45019eb51f8d75497748 (127.7.17.195:43415)
I20260927 15:07:57.357191 15441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:45121
I20260927 15:07:57.365818 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.7.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.193.keytab.
Entry for principal oryx/127.7.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.193.keytab.
WARNING: no policy specified for oryx/127.7.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.194.keytab.
Entry for principal oryx/127.7.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.194.keytab.
WARNING: no policy specified for oryx/127.7.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.195.keytab.
Entry for principal oryx/127.7.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.195.keytab.
I20260927 15:07:57.439599 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 15503
I20260927 15:07:57.444651 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 15637
I20260927 15:07:57.449627 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 15771
I20260927 15:07:57.454334 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 15409
I20260927 15:07:57.459673 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:46207
--webserver_interface=127.7.17.254
--webserver_port=33385
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:46207
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:07:57.567458 15909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:57.567773 15909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:57.567834 15909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:57.572084 15909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:07:57.572160 15909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:57.572183 15909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:57.572206 15909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:07:57.572237 15909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:07:57.577821 15909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:46207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:46207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=33385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:57.578954 15909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:57.579978 15909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:57.588614 15915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:57.588657 15914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:57.588657 15917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:57.589047 15909 server_base.cc:1034] running on GCE node
I20260927 15:07:57.589648 15909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:57.590852 15909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:57.592087 15909 hybrid_clock.cc:648] HybridClock initialized: now 1790521677592059 us; error 43 us; skew 500 ppm
Sep 27 15:07:57 dist-test-slave-hhpq krb5kdc[15396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521677, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:57.595110 15909 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:07:57.596033 15909 webserver.cc:460] Webserver started at http://127.7.17.254:33385/ using document root <none> and password file <none>
I20260927 15:07:57.596575 15909 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:57.596647 15909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:57.600801 15909 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:57.603053 15924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:57.604334 15909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:07:57.604451 15909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "6eb96c7b7a1241198fc952e9f163e983"
format_stamp: "Formatted at 2026-09-27 15:07:56 on dist-test-slave-hhpq"
server_key: "12f66a00bbcc5d322f0fccb99e2f6585"
server_key_iv: "2524f5a2876ebcfdbe381ab77c9b8cf1"
server_key_version: "encryptionkey@0"
I20260927 15:07:57.605052 15909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:57.617522 15909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:57.620676 15909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:57.620882 15909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:57.627434 15909 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:46207
I20260927 15:07:57.627494 15976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:46207 every 8 connection(s)
I20260927 15:07:57.628504 15909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:07:57.632966 15977 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:07:57.634707 15977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983: Bootstrap starting.
I20260927 15:07:57.635747 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 15909
I20260927 15:07:57.636224 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:42371
--local_ip_for_outbound_sockets=127.7.17.193
--tserver_master_addrs=127.7.17.254:46207
--webserver_port=33809
--webserver_interface=127.7.17.193
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.17.193
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:07:57.651666 15977 log.cc:824] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983: Log is configured to *not* fsync() on all Append() calls
I20260927 15:07:57.658488 15977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:07:57.658835 15977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983: Bootstrap complete.
I20260927 15:07:57.662972 15977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb96c7b7a1241198fc952e9f163e983" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46207 } }
I20260927 15:07:57.663398 15977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eb96c7b7a1241198fc952e9f163e983, State: Initialized, Role: FOLLOWER
I20260927 15:07:57.663898 15977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb96c7b7a1241198fc952e9f163e983" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46207 } }
I20260927 15:07:57.664024 15977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:07:57.664089 15977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:07:57.664193 15977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:07:57.666170 15977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb96c7b7a1241198fc952e9f163e983" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46207 } }
I20260927 15:07:57.666501 15977 leader_election.cc:302] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6eb96c7b7a1241198fc952e9f163e983; no voters:
I20260927 15:07:57.666817 15977 leader_election.cc:288] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:07:57.666916 15981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:07:57.667101 15981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [term 2 LEADER]: Becoming Leader. State: Replica: 6eb96c7b7a1241198fc952e9f163e983, State: Running, Role: LEADER
I20260927 15:07:57.667394 15981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb96c7b7a1241198fc952e9f163e983" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46207 } }
I20260927 15:07:57.668089 15977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:07:57.668347 15981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6eb96c7b7a1241198fc952e9f163e983. Latest consensus state: current_term: 2 leader_uuid: "6eb96c7b7a1241198fc952e9f163e983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb96c7b7a1241198fc952e9f163e983" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46207 } } }
I20260927 15:07:57.668458 15981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:57.668709 15982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6eb96c7b7a1241198fc952e9f163e983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb96c7b7a1241198fc952e9f163e983" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46207 } } }
I20260927 15:07:57.668843 15982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983 [sys.catalog]: This master's current role is: LEADER
I20260927 15:07:57.669137 15985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:07:57.673621 15985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:07:57.674283 15985 catalog_manager.cc:1259] Loaded cluster ID: 91d41833e8ae4450bc8fdb1f391cc2d1
I20260927 15:07:57.674346 15985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:07:57.676754 15985 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:07:57.677502 15985 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6eb96c7b7a1241198fc952e9f163e983: Loaded TSK: 0
I20260927 15:07:57.678807 15985 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:07:57.752375 15979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:57.752749 15979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:57.752884 15979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:57.757269 15979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:57.757356 15979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:57.757443 15979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:07:57.763026 15979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.193.keytab
--principal=oryx/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:42371
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=33809
--tserver_master_addrs=127.7.17.254:46207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:57.764151 15979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:57.765272 15979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:57.776283 16006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:57.776311 16004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:57.776525 16003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:57.776770 15979 server_base.cc:1034] running on GCE node
I20260927 15:07:57.777235 15979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:57.777887 15979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:57.779068 15979 hybrid_clock.cc:648] HybridClock initialized: now 1790521677779032 us; error 53 us; skew 500 ppm
Sep 27 15:07:57 dist-test-slave-hhpq krb5kdc[15396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521677, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:57.782074 15979 init.cc:377] Logged in from keytab as oryx/127.7.17.193@KRBTEST.COM (short username oryx)
I20260927 15:07:57.783061 15979 webserver.cc:460] Webserver started at http://127.7.17.193:33809/ using document root <none> and password file <none>
I20260927 15:07:57.783648 15979 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:57.783735 15979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:57.787933 15979 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:07:57.790123 16013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:57.791383 15979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:07:57.791498 15979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "496d63d5525e43e4a6a548e8dcf4cba8"
format_stamp: "Formatted at 2026-09-27 15:07:56 on dist-test-slave-hhpq"
server_key: "02058f8692ad23da35ed65a1ebaf2a1d"
server_key_iv: "83f720547c79fce62d5a2cf5614204f3"
server_key_version: "encryptionkey@0"
I20260927 15:07:57.792032 15979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:57.817124 15979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:57.820081 15979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:57.820264 15979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:57.820883 15979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:57.821713 15979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:57.821760 15979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:57.821798 15979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:57.821815 15979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:57.830540 15979 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:42371
I20260927 15:07:57.830615 16126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:42371 every 8 connection(s)
I20260927 15:07:57.831579 15979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:07:57.833014 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 15979
I20260927 15:07:57.833455 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:44801
--local_ip_for_outbound_sockets=127.7.17.194
--tserver_master_addrs=127.7.17.254:46207
--webserver_port=39043
--webserver_interface=127.7.17.194
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.17.194
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:07:57 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:57 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:57.847393 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.833863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.193:47141)
0927 15:07:57.835477 (+ 1614us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:57.835539 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:57.836536 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:57.837806 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:57.837815 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:57.838309 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:57.838995 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.839017 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.840400 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.840403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.841876 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.841889 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.842683 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.842687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.842888 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.843526 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:57.843548 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:57.846295 (+ 2747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1399,"thread_start_us":593,"threads_started":1}
W20260927 15:07:57.847386 16129 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:57.833774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:47141 (local address 127.7.17.254:46207)
0927 15:07:57.834321 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:57.834340 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:57.835667 (+ 1327us) server_negotiation.cc:407] Connection header received
0927 15:07:57.836991 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:57.837003 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:57.837355 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:57.837638 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:57.839206 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.840250 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.842020 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.842504 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.842646 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.846568 (+ 3922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.193:47141: BlockingRecv error: recv got EOF from 127.7.17.193:47141 (error 108)
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":136,"threads_started":1}
W20260927 15:07:57.847931 16127 heartbeater.cc:644] Failed to heartbeat to 127.7.17.254:46207 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.17.254:46207: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Sep 27 15:07:57 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:57 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:57.853852 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.848403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.193:45811)
0927 15:07:57.848575 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:57.848589 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:57.848673 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:57.848917 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:57.848920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:57.849091 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:57.849203 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.849208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.849931 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.849935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.850832 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.850841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.851287 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.851291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.851381 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.851833 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:57.851847 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:57.853711 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106}
W20260927 15:07:57.853989 16129 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:57.848493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:45811 (local address 127.7.17.254:46207)
0927 15:07:57.848595 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:57.848598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:57.848629 (+ 31us) server_negotiation.cc:407] Connection header received
0927 15:07:57.848752 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:57.848754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:57.848794 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:57.848892 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:57.849319 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.849829 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.850971 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.851187 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.851279 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.853862 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.193:45811: BlockingRecv error: recv got EOF from 127.7.17.193:45811 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 15:07:57 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:57 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:57.859485 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:57.854363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.193:34201)
0927 15:07:57.854698 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:57.854710 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:57.854803 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:57.855059 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:57.855062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:57.855246 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:57.855334 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.855339 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.856034 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.856037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.856623 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:57.856631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.857120 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.857123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:57.857223 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.857697 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:57.857709 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:57.859304 (+ 1595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":107,"threads_started":1}
W20260927 15:07:57.859535 16129 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:57.854439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:34201 (local address 127.7.17.254:46207)
0927 15:07:57.854565 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:57.854569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:57.854775 (+ 206us) server_negotiation.cc:407] Connection header received
0927 15:07:57.854880 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:57.854883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:57.854929 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:57.855047 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:57.855439 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.855928 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.856729 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:57.857005 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:57.857095 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:57.859438 (+ 2343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.193:34201: BlockingRecv error: recv got EOF from 127.7.17.193:34201 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 15:07:57.859745 16127 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:07:57.949856 16130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:57.950155 16130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:57.950202 16130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:57.954357 16130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:57.954434 16130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:57.954495 16130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:07:57.959975 16130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.194.keytab
--principal=oryx/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:44801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=39043
--tserver_master_addrs=127.7.17.254:46207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:57.961123 16130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:57.962227 16130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:57.972090 16140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:57.972095 16138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:57.972193 16130 server_base.cc:1034] running on GCE node
W20260927 15:07:57.972093 16137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:57.972733 16130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:57.973444 16130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:57.974639 16130 hybrid_clock.cc:648] HybridClock initialized: now 1790521677974608 us; error 44 us; skew 500 ppm
Sep 27 15:07:57 dist-test-slave-hhpq krb5kdc[15396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521677, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:57.977483 16130 init.cc:377] Logged in from keytab as oryx/127.7.17.194@KRBTEST.COM (short username oryx)
I20260927 15:07:57.978366 16130 webserver.cc:460] Webserver started at http://127.7.17.194:39043/ using document root <none> and password file <none>
I20260927 15:07:57.978900 16130 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:57.978972 16130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:57.983026 16130 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:07:57.985316 16147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:57.986726 16130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:07:57.986838 16130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "1f88d69e05e146bb80e3076425201a98"
format_stamp: "Formatted at 2026-09-27 15:07:57 on dist-test-slave-hhpq"
server_key: "b543218da1fddab7b5aae90db448a832"
server_key_iv: "d4bbb7fa00b729ae6f0ce49ffaa7b0d3"
server_key_version: "encryptionkey@0"
I20260927 15:07:57.987408 16130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:58.006650 16130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:58.010032 16130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:58.010219 16130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:58.010812 16130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:58.011636 16130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:58.011682 16130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:58.011719 16130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:58.011745 16130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:58.020653 16130 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:44801
I20260927 15:07:58.020716 16260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:44801 every 8 connection(s)
I20260927 15:07:58.021759 16130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:07:58.030570 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 16130
I20260927 15:07:58.031004 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:43415
--local_ip_for_outbound_sockets=127.7.17.195
--tserver_master_addrs=127.7.17.254:46207
--webserver_port=44833
--webserver_interface=127.7.17.195
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.17.195
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:07:58 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:58 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 15:07:58.033143 16129 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:58.023722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:38339 (local address 127.7.17.254:46207)
0927 15:07:58.023842 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:58.023846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:58.024762 (+ 916us) server_negotiation.cc:407] Connection header received
0927 15:07:58.026083 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:58.026086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:58.026142 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:58.026220 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:58.027481 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.028091 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.029289 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.029516 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.029606 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:58.033011 (+ 3405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.194:38339: BlockingRecv error: recv got EOF from 127.7.17.194:38339 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:07:58.033757 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:58.024138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.194:38339)
0927 15:07:58.024591 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:58.024635 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:58.025833 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:58.026382 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:58.026389 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:58.026740 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:58.027320 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:58.027336 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.028199 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.028201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:58.029156 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:58.029165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.029642 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.029645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:58.029787 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:58.030362 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:58.030379 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:58.032774 (+ 2395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":114,"threads_started":1}
W20260927 15:07:58.034226 16261 heartbeater.cc:644] Failed to heartbeat to 127.7.17.254:46207 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.17.254:46207: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Sep 27 15:07:58 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:58 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:58.039834 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:58.034667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.194:46895)
0927 15:07:58.034776 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:58.034872 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:58.035010 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:58.035242 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:58.035245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:58.035395 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:58.035489 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:58.035495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.036196 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.036199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:58.036746 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:58.036755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.037190 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.037194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:58.037277 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:58.037766 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:58.037777 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:58.039701 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 15:07:58.039956 16129 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:58.034749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:46895 (local address 127.7.17.254:46207)
0927 15:07:58.034849 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:58.034852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:58.034930 (+ 78us) server_negotiation.cc:407] Connection header received
0927 15:07:58.035108 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:58.035111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:58.035153 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:58.035216 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:58.035597 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.036097 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.036878 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.037092 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.037181 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:58.039854 (+ 2673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.194:46895: BlockingRecv error: recv got EOF from 127.7.17.194:46895 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 15:07:58 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:58 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:58.045352 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:58.040554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.194:43071)
0927 15:07:58.040664 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:58.040678 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:58.040757 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:58.041002 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:58.041005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:58.041146 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:58.041221 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:58.041226 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.041920 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.041923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:58.042439 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:58.042446 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.042854 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.042857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:58.042930 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:58.043451 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:58.043464 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:58.045221 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 15:07:58.045480 16129 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:58.040627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:43071 (local address 127.7.17.254:46207)
0927 15:07:58.040730 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:58.040734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:58.040746 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:07:58.040867 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:58.040870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:58.040912 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:58.040975 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:58.041324 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.041824 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.042546 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.042760 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.042844 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:58.045354 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.194:43071: BlockingRecv error: recv got EOF from 127.7.17.194:43071 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 15:07:58.045589 16261 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:07:58.141716 16264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:07:58.142009 16264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:07:58.142069 16264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:07:58.146194 16264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:07:58.146270 16264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:07:58.146354 16264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:07:58.151793 16264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/krb5kdc/oryx_127.7.17.195.keytab
--principal=oryx/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:43415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=44833
--tserver_master_addrs=127.7.17.254:46207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:07:58.152942 16264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:07:58.153992 16264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:07:58.163544 16269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:58.163583 16264 server_base.cc:1034] running on GCE node
W20260927 15:07:58.163514 16270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:07:58.163514 16272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:07:58.164245 16264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:07:58.164892 16264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:07:58.166085 16264 hybrid_clock.cc:648] HybridClock initialized: now 1790521678166065 us; error 37 us; skew 500 ppm
Sep 27 15:07:58 dist-test-slave-hhpq krb5kdc[15396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521678, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:07:58.169229 16264 init.cc:377] Logged in from keytab as oryx/127.7.17.195@KRBTEST.COM (short username oryx)
I20260927 15:07:58.170202 16264 webserver.cc:460] Webserver started at http://127.7.17.195:44833/ using document root <none> and password file <none>
I20260927 15:07:58.170787 16264 fs_manager.cc:359] Metadata directory not provided
I20260927 15:07:58.170861 16264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:07:58.174965 16264 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 15:07:58.177204 16279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:58.178666 16264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:07:58.178783 16264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "2ee353359b4f45019eb51f8d75497748"
format_stamp: "Formatted at 2026-09-27 15:07:57 on dist-test-slave-hhpq"
server_key: "611b2dfe2a29b4c5c52e7df4862ab4a6"
server_key_iv: "2296917883b3d3480e1c3ba7aa4022f6"
server_key_version: "encryptionkey@0"
I20260927 15:07:58.179324 16264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:07:58.194905 16264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:07:58.198032 16264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:07:58.198228 16264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:07:58.198805 16264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:07:58.199589 16264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:07:58.199635 16264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:58.199674 16264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:07:58.199694 16264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:07:58.207947 16264 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:43415
I20260927 15:07:58.208015 16392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:43415 every 8 connection(s)
I20260927 15:07:58.209100 16264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:07:58.217422 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 16264
Sep 27 15:07:58 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:58 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 15:07:58.220184 16129 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:58.210930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:36823 (local address 127.7.17.254:46207)
0927 15:07:58.211071 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:58.211075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:58.211897 (+ 822us) server_negotiation.cc:407] Connection header received
0927 15:07:58.213221 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:58.213225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:58.213285 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:58.213359 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:58.214570 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.215094 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.216281 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.216497 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.216585 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:58.220045 (+ 3460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.195:36823: BlockingRecv error: recv got EOF from 127.7.17.195:36823 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:07:58.220878 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:58.211322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.195:36823)
0927 15:07:58.211742 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:58.211787 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:58.213007 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:58.213571 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:58.213579 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:58.213933 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:58.214412 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:58.214426 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.215218 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.215221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:58.216155 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:58.216165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.216640 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.216645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:58.216809 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:58.217467 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:58.217486 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:58.219909 (+ 2423us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":96,"threads_started":1}
W20260927 15:07:58.221431 16393 heartbeater.cc:644] Failed to heartbeat to 127.7.17.254:46207 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.17.254:46207: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Sep 27 15:07:58 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:58 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:58.226984 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:58.221846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.195:57415)
0927 15:07:58.221968 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:58.221982 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:58.222057 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:58.222329 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:58.222331 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:58.222534 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:58.222648 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:58.222655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.223366 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.223369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:58.224110 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:58.224122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.224645 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.224649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:58.224742 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:58.225145 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:58.225160 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:58.226819 (+ 1659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 15:07:58.227104 16129 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:58.221921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:57415 (local address 127.7.17.254:46207)
0927 15:07:58.222027 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:58.222030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:58.222044 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:07:58.222144 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:58.222147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:58.222191 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:58.222270 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:58.222757 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.223259 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.224234 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.224539 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.224615 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:58.226948 (+ 2333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.195:57415: BlockingRecv error: recv got EOF from 127.7.17.195:57415 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:07:58.227689 16397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:58.218413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58540 (local address 127.7.17.254:46207)
0927 15:07:58.218631 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:58.218635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:58.218649 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:07:58.218683 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:58.218686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:58.218724 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:58.218803 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:58.219555 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.220173 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.221406 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.221697 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.221793 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:58.223346 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:07:58.223353 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:07:58.223373 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:07:58.223406 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:07:58.225490 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:58.225950 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:58.225957 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:58.225964 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:58.226105 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:07:58.226359 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:07:58.226362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:07:58.226364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:07:58.226753 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:07:58.226934 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 15:07:58.227292 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 15:07:58.227567 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":75,"threads_started":1}
Sep 27 15:07:58 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:58 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:58.232569 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:58.227848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.195:53835)
0927 15:07:58.228020 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:58.228035 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:58.228108 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:58.228389 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:58.228390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:58.228504 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:58.228580 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:58.228584 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.229331 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.229335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:58.229978 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:58.229987 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.230405 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.230409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:58.230479 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:58.230855 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:58.230874 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:58.232472 (+ 1598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 15:07:58.232697 16129 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:58.228074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:53835 (local address 127.7.17.254:46207)
0927 15:07:58.228212 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:58.228215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:58.228227 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:07:58.228254 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:58.228257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:58.228302 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:58.228374 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:58.228672 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.229234 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.230063 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.230300 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.230379 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:58.232594 (+ 2215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.195:53835: BlockingRecv error: recv got EOF from 127.7.17.195:53835 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 15:07:58.232770 16393 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 15:07:58 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:58 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:58.869650 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:58.860581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.193:35761)
0927 15:07:58.860882 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:58.860952 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:58.861067 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:58.861327 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:58.861329 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:58.861502 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:58.861601 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:58.861606 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.862393 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.862396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:58.863040 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:58.863050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.863502 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.863505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:58.863576 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:58.867407 (+ 3831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:58.867423 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:58.869504 (+ 2081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":77,"threads_started":1}
W20260927 15:07:58.869824 16399 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:58.860667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:35761 (local address 127.7.17.254:46207)
0927 15:07:58.860973 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:58.860977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:58.861020 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:07:58.861159 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:58.861161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:58.861205 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:58.861288 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:58.861720 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.862263 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.863172 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:58.863404 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:58.863487 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:58.869692 (+ 6205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.193:35761: BlockingRecv error: recv got EOF from 127.7.17.193:35761 (error 108)
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":91,"threads_started":1}
Sep 27 15:07:59 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:59 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:59.051950 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:59.046264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.194:52099)
0927 15:07:59.046519 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:59.046533 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:59.046627 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:59.046988 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:59.046991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:59.047171 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:59.047269 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:59.047274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:59.048062 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:59.048065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:59.048634 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:59.048642 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:59.049157 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:59.049160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:59.049231 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:59.049764 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:59.049776 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:59.051761 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":76,"threads_started":1}
W20260927 15:07:59.052042 16399 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:59.046404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:52099 (local address 127.7.17.254:46207)
0927 15:07:59.046563 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:59.046566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:59.046580 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:07:59.046752 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:59.046755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:59.046805 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:59.046888 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:59.047379 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:59.047902 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:59.048754 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:59.049028 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:59.049115 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:59.051942 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.194:52099: BlockingRecv error: recv got EOF from 127.7.17.194:52099 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 15:07:59 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:59 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:59.238979 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:59.233495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.195:59585)
0927 15:07:59.233765 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:59.233780 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:59.233892 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:59.234155 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:59.234158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:59.234358 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:59.234476 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:59.234484 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:59.235319 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:59.235322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:59.235901 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:59.235908 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:59.236344 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:59.236348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:59.236443 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:59.237084 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:59.237096 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:59.238842 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":94,"threads_started":1}
W20260927 15:07:59.239074 16399 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:59.233573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:59585 (local address 127.7.17.254:46207)
0927 15:07:59.233699 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:59.233703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:59.233849 (+ 146us) server_negotiation.cc:407] Connection header received
0927 15:07:59.233973 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:59.233976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:59.234042 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:59.234120 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:59.234640 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:59.235215 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:59.236005 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:59.236219 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:59.236298 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:59.238970 (+ 2672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.195:59585: BlockingRecv error: recv got EOF from 127.7.17.195:59585 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 15:07:59 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:07:59 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:07:59.877038 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:07:59.870515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.193:44683)
0927 15:07:59.870777 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 15:07:59.870792 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:07:59.870887 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:07:59.871231 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:07:59.871234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:07:59.871404 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:07:59.871505 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:59.871510 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:59.872378 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:59.872381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:59.873060 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:07:59.873072 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:07:59.873494 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:07:59.873496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:07:59.873554 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:59.874219 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:07:59.874232 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:07:59.876852 (+ 2620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
W20260927 15:07:59.877840 16403 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:07:59.870592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:44683 (local address 127.7.17.254:46207)
0927 15:07:59.870864 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 15:07:59.870867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:07:59.870883 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:07:59.871023 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:07:59.871026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:07:59.871070 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:07:59.871158 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:07:59.871660 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:59.872244 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:59.873175 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:07:59.873403 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:07:59.873483 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:07:59.877633 (+ 4150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.193:44683: BlockingRecv error: recv got EOF from 127.7.17.193:44683 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":96,"threads_started":1}
Sep 27 15:08:00 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:00 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:00.057899 16404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:00.052740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.194:38337)
0927 15:08:00.053022 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:00.053036 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:00.053134 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:00.053352 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:00.053354 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:00.053526 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:00.053624 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:00.053629 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:00.054336 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:00.054338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:00.054878 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:00.054884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:00.055322 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:00.055326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:00.055402 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:00.055963 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:00.055976 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:00.057741 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":97,"threads_started":1}
W20260927 15:08:00.058014 16403 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:00.052882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:38337 (local address 127.7.17.254:46207)
0927 15:08:00.053017 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:00.053021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:00.053098 (+ 77us) server_negotiation.cc:407] Connection header received
0927 15:08:00.053207 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:00.053209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:00.053251 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:00.053330 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:00.053734 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:00.054251 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:00.054962 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:00.055188 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:00.055266 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:00.057895 (+ 2629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.194:38337: BlockingRecv error: recv got EOF from 127.7.17.194:38337 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 15:08:00 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:00 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:00.245124 16405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:00.239728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.195:49297)
0927 15:08:00.240068 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:00.240085 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:00.240194 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:00.240447 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:00.240450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:00.240707 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:00.240807 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:00.240813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:00.241608 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:00.241610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:00.242161 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:00.242167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:00.242546 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:00.242548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:00.242604 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:00.243114 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:00.243126 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:00.244963 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":91,"threads_started":1}
W20260927 15:08:00.245229 16403 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:00.239850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:49297 (local address 127.7.17.254:46207)
0927 15:08:00.239972 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:00.239976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:00.240164 (+ 188us) server_negotiation.cc:407] Connection header received
0927 15:08:00.240283 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:00.240286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:00.240335 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:00.240399 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:00.240972 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:00.241493 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:00.242248 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:00.242464 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:00.242562 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:00.245129 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.195:49297: BlockingRecv error: recv got EOF from 127.7.17.195:49297 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 15:08:00 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:00 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:00.883785 16406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:00.878081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.193:44767)
0927 15:08:00.878348 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:00.878363 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:00.878473 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:00.878764 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:00.878766 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:00.878998 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:00.879098 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:00.879104 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:00.879966 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:00.879969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:00.880532 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:00.880541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:00.881095 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:00.881098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:00.881172 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:00.881669 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:00.881681 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:00.883614 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":84,"threads_started":1}
W20260927 15:08:00.883978 16407 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:00.878197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:44767 (local address 127.7.17.254:46207)
0927 15:08:00.878427 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:00.878431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:00.878445 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:08:00.878555 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:00.878558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:00.878598 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:00.878712 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:00.879231 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:00.879812 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:00.880645 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:00.880976 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:00.881064 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:00.883831 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.193:44767: BlockingRecv error: recv got EOF from 127.7.17.193:44767 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":71,"threads_started":1}
Sep 27 15:08:01 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:01 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:01.064270 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:01.058789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.194:57007)
0927 15:08:01.059045 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:01.059061 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:01.059155 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:01.059393 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:01.059395 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:01.059590 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:01.059700 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:01.059706 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:01.060506 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:01.060509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:01.061190 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:01.061202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:01.061620 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:01.061623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:01.061683 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:01.062179 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:01.062191 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:01.064144 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
W20260927 15:08:01.064446 16407 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:01.058958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:57007 (local address 127.7.17.254:46207)
0927 15:08:01.059091 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:01.059095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:01.059110 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:08:01.059238 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:01.059240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:01.059283 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:01.059361 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:01.059848 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:01.060370 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:01.061286 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:01.061514 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:01.061603 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:01.064319 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.194:57007: BlockingRecv error: recv got EOF from 127.7.17.194:57007 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 15:08:01 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:01 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:01.251026 16409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:01.245870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.195:39009)
0927 15:08:01.246201 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:01.246216 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:01.246334 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:01.246558 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:01.246560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:01.246730 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:01.246830 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:01.246835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:01.247591 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:01.247595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:01.248147 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:01.248155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:01.248556 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:01.248558 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:01.248620 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:01.249209 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:01.249221 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:01.250909 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":146,"threads_started":1}
W20260927 15:08:01.251230 16407 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:01.245960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:39009 (local address 127.7.17.254:46207)
0927 15:08:01.246105 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:01.246109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:01.246299 (+ 190us) server_negotiation.cc:407] Connection header received
0927 15:08:01.246412 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:01.246414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:01.246464 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:01.246526 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:01.246962 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:01.247491 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:01.248243 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:01.248468 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:01.248541 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:01.251109 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.195:39009: BlockingRecv error: recv got EOF from 127.7.17.195:39009 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 15:08:01 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:01 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:01.890842 16410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:01.884670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.193:48947)
0927 15:08:01.885011 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:01.885029 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:01.885163 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:01.885418 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:01.885421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:01.885641 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:01.885739 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:01.885745 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:01.886605 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:01.886608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:01.887431 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:01.887444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:01.888023 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:01.888027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:01.888100 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:01.888659 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:01.888672 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:01.890716 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":95,"threads_started":1}
W20260927 15:08:01.891036 16411 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:01.884764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:48947 (local address 127.7.17.254:46207)
0927 15:08:01.885018 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:01.885024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:01.885112 (+ 88us) server_negotiation.cc:407] Connection header received
0927 15:08:01.885259 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:01.885262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:01.885308 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:01.885384 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:01.885870 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:01.886451 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:01.887540 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:01.887846 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:01.887948 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:01.890895 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.193:48947: BlockingRecv error: recv got EOF from 127.7.17.193:48947 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":84,"threads_started":1}
Sep 27 15:08:02 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:02 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:02.071255 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:02.065195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.194:42947)
0927 15:08:02.065504 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:02.065520 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:02.065647 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:02.065892 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:02.065895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:02.066148 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:02.066255 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:02.066262 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:02.067023 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:02.067026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:02.067732 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:02.067746 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:02.068354 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:02.068358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:02.068456 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:02.069167 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:02.069184 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:02.071133 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":102,"threads_started":1}
W20260927 15:08:02.071391 16411 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:02.065301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:42947 (local address 127.7.17.254:46207)
0927 15:08:02.065482 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:02.065487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:02.065606 (+ 119us) server_negotiation.cc:407] Connection header received
0927 15:08:02.065738 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:02.065741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:02.065789 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:02.065861 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:02.066380 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:02.066903 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:02.067966 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:02.068242 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:02.068337 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:02.071281 (+ 2944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.194:42947: BlockingRecv error: recv got EOF from 127.7.17.194:42947 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 15:08:02 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:02 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:02.257206 16413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:02.251806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.195:37471)
0927 15:08:02.252112 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:02.252124 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:02.252224 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:02.252503 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:02.252505 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:02.252677 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:02.252768 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:02.252773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:02.253513 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:02.253517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:02.254128 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:02.254138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:02.254619 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:02.254622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:02.254692 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:02.255177 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:02.255188 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:02.257086 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":70,"threads_started":1}
W20260927 15:08:02.257347 16411 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:02.251928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:37471 (local address 127.7.17.254:46207)
0927 15:08:02.252057 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:02.252061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:02.252185 (+ 124us) server_negotiation.cc:407] Connection header received
0927 15:08:02.252334 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:02.252337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:02.252388 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:02.252463 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:02.252874 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:02.253400 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:02.254255 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:02.254490 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:02.254582 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:02.257224 (+ 2642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.195:37471: BlockingRecv error: recv got EOF from 127.7.17.195:37471 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 15:08:02 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:02 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:02.897310 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:02.891637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.193:45009)
0927 15:08:02.891932 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:02.891948 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:02.892054 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:02.892303 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:02.892304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:02.892511 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:02.892604 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:02.892610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:02.893470 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:02.893472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:02.894047 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:02.894056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:02.894516 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:02.894519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:02.894585 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:02.895121 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:02.895133 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:02.897152 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":75,"threads_started":1}
W20260927 15:08:02.897410 16415 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:02.891810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:45009 (local address 127.7.17.254:46207)
0927 15:08:02.892055 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:02.892058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:02.892073 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:08:02.892141 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:02.892144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:02.892192 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:02.892265 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:02.892719 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:02.893343 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:02.894171 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:02.894401 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:02.894500 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:02.897296 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.193:45009: BlockingRecv error: recv got EOF from 127.7.17.193:45009 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":72,"threads_started":1}
Sep 27 15:08:03 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:03 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:03.077862 16416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:03.072160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.194:48313)
0927 15:08:03.072459 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:03.072474 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:03.072570 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:03.072792 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:03.072794 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:03.073093 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:03.073204 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:03.073212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:03.074001 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:03.074005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:03.074602 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:03.074612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:03.075104 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:03.075108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:03.075191 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:03.075794 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:03.075810 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:03.077739 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":76,"threads_started":1}
W20260927 15:08:03.077961 16415 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:03.072369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:48313 (local address 127.7.17.254:46207)
0927 15:08:03.072507 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:03.072510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:03.072528 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:08:03.072654 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:03.072656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:03.072696 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:03.072765 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:03.073323 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:03.073865 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:03.074740 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:03.074968 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:03.075068 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:03.077864 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.194:48313: BlockingRecv error: recv got EOF from 127.7.17.194:48313 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 15:08:03 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:03 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:03.262881 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:03.257985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.195:40821)
0927 15:08:03.258234 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:03.258247 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:03.258336 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:03.258556 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:03.258558 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:03.258725 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:03.258814 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:03.258819 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:03.259533 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:03.259535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:03.260127 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:03.260135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:03.260552 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:03.260555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:03.260620 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:03.261100 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:03.261111 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:03.262759 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":65,"threads_started":1}
W20260927 15:08:03.263028 16415 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:03.258128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:40821 (local address 127.7.17.254:46207)
0927 15:08:03.258271 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:03.258275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:03.258291 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:08:03.258414 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:03.258416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:03.258459 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:03.258528 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:03.258894 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:03.259438 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:03.260232 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:03.260452 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:03.260536 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:03.262894 (+ 2358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.195:40821: BlockingRecv error: recv got EOF from 127.7.17.195:40821 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 15:08:03 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:03 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:03.908206 16418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:03.898142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.193:35061)
0927 15:08:03.898441 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:03.898457 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:03.898558 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:03.898917 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:03.898920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:03.899121 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:03.899220 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:03.899226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:03.900108 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:03.900112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:03.900717 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:03.900728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:03.901341 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:03.901345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:03.901437 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:03.905954 (+ 4517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:03.905972 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:03.908017 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":87,"threads_started":1}
W20260927 15:08:03.908324 16419 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:03.898276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:35061 (local address 127.7.17.254:46207)
0927 15:08:03.898510 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:03.898514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:03.898530 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:08:03.898687 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:03.898690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:03.898750 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:03.898838 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:03.899332 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:03.899873 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:03.900942 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:03.901219 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:03.901310 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:03.908201 (+ 6891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.193:35061: BlockingRecv error: recv got EOF from 127.7.17.193:35061 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":53,"threads_started":1}
Sep 27 15:08:04 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:04 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:04.084218 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:04.078682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.194:44299)
0927 15:08:04.079058 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:04.079074 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:04.079181 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:04.079390 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:04.079392 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:04.079561 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:04.079655 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:04.079660 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:04.080409 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:04.080412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:04.081016 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:04.081026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:04.081443 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:04.081445 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:04.081516 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:04.082042 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:04.082055 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:04.084064 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
W20260927 15:08:04.084306 16419 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:04.078730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:44299 (local address 127.7.17.254:46207)
0927 15:08:04.078877 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:04.078881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:04.079131 (+ 250us) server_negotiation.cc:407] Connection header received
0927 15:08:04.079262 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:04.079264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:04.079310 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:04.079366 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:04.079747 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:04.080294 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:04.081132 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:04.081351 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:04.081429 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:04.084206 (+ 2777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.194:44299: BlockingRecv error: recv got EOF from 127.7.17.194:44299 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 15:08:04 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:04 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:04.269184 16421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:04.263712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.195:54669)
0927 15:08:04.264032 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:04.264046 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:04.264160 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:04.264407 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:04.264410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:04.264622 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:04.264716 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:04.264722 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:04.265505 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:04.265508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:04.266062 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:04.266070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:04.266496 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:04.266500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:04.266580 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:04.267062 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:04.267074 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:04.269035 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":94,"threads_started":1}
W20260927 15:08:04.269212 16419 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:04.263840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:54669 (local address 127.7.17.254:46207)
0927 15:08:04.263985 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:04.263990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:04.264120 (+ 130us) server_negotiation.cc:407] Connection header received
0927 15:08:04.264248 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:04.264252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:04.264300 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:04.264371 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:04.264868 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:04.265393 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:04.266174 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:04.266395 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:04.266489 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:04.269117 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.195:54669: BlockingRecv error: recv got EOF from 127.7.17.195:54669 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 15:08:04 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:04 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:04.914324 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:04.909040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.193:44347)
0927 15:08:04.909339 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:04.909353 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:04.909470 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:04.909703 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:04.909705 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:04.909878 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:04.910016 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:04.910023 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:04.910717 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:04.910719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:04.911315 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:04.911323 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:04.911777 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:04.911779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:04.911842 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:04.912358 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:04.912372 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:04.914164 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":86,"threads_started":1}
W20260927 15:08:04.914436 16423 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:04.909118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:44347 (local address 127.7.17.254:46207)
0927 15:08:04.909339 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:04.909342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:04.909401 (+ 59us) server_negotiation.cc:407] Connection header received
0927 15:08:04.909559 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:04.909561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:04.909602 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:04.909676 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:04.910101 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:04.910639 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:04.911442 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:04.911676 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:04.911764 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:04.914322 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.193:44347: BlockingRecv error: recv got EOF from 127.7.17.193:44347 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":61,"threads_started":1}
Sep 27 15:08:05 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:05 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:05.090368 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:05.085013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.194:54313)
0927 15:08:05.085289 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:05.085304 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:05.085420 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:05.085660 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:05.085663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:05.085863 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:05.085964 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:05.085970 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:05.086702 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:05.086706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:05.087327 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:05.087338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:05.087764 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:05.087768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:05.087847 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:05.088344 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:05.088363 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:05.090217 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":77,"threads_started":1}
W20260927 15:08:05.090629 16423 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:05.085063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:54313 (local address 127.7.17.254:46207)
0927 15:08:05.085193 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:05.085197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:05.085406 (+ 209us) server_negotiation.cc:407] Connection header received
0927 15:08:05.085501 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:05.085504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:05.085554 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:05.085618 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:05.086075 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:05.086598 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:05.087445 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:05.087664 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:05.087750 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:05.090408 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.194:54313: BlockingRecv error: recv got EOF from 127.7.17.194:54313 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 15:08:05 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:05 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:05.275470 16425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:05.270015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.195:52891)
0927 15:08:05.270330 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:05.270345 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:05.270459 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:05.270724 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:05.270727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:05.270894 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:05.271000 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:05.271005 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:05.271765 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:05.271768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:05.272560 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:05.272571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:05.273018 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:05.273021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:05.273086 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:05.273622 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:05.273636 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:05.275350 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":87,"threads_started":1}
W20260927 15:08:05.275624 16423 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:05.270104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:52891 (local address 127.7.17.254:46207)
0927 15:08:05.270305 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:05.270308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:05.270420 (+ 112us) server_negotiation.cc:407] Connection header received
0927 15:08:05.270536 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:05.270539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:05.270586 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:05.270657 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:05.271101 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:05.271672 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:05.272673 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:05.272929 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:05.273005 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:05.275492 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.195:52891: BlockingRecv error: recv got EOF from 127.7.17.195:52891 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 27 15:08:05 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:05 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:05.921368 16426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:05.915366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.193:45145)
0927 15:08:05.915764 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:05.915780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:05.915877 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:05.916160 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:05.916162 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:05.916331 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:05.916436 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:05.916442 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:05.917312 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:05.917316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:05.918129 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:05.918142 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:05.918724 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:05.918726 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:05.918794 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:05.919305 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:05.919319 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:05.921219 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":99,"threads_started":1}
W20260927 15:08:05.921510 16427 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:05.915588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:45145 (local address 127.7.17.254:46207)
0927 15:08:05.915853 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:05.915856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:05.915870 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:08:05.916004 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:05.916006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:05.916050 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:05.916137 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:05.916539 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:05.917183 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:05.918275 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:05.918607 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:05.918700 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:05.921362 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.193:45145: BlockingRecv error: recv got EOF from 127.7.17.193:45145 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":88,"threads_started":1}
Sep 27 15:08:06 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:06 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:06.097083 16428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:06.091179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.194:49129)
0927 15:08:06.091425 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:06.091441 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:06.091577 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:06.091933 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:06.091935 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:06.092146 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:06.092254 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:06.092260 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:06.093262 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:06.093265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:06.093833 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:06.093842 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:06.094277 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:06.094280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:06.094348 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:06.094898 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:06.094911 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:06.096967 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":75,"threads_started":1}
W20260927 15:08:06.097242 16427 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:06.091262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:49129 (local address 127.7.17.254:46207)
0927 15:08:06.091410 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:06.091413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:06.091513 (+ 100us) server_negotiation.cc:407] Connection header received
0927 15:08:06.091704 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:06.091707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:06.091759 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:06.091900 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:06.092390 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:06.093160 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:06.093938 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:06.094167 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:06.094248 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:06.097119 (+ 2871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.194:49129: BlockingRecv error: recv got EOF from 127.7.17.194:49129 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 15:08:06 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:06 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:06.281428 16429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:06.276212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.195:43171)
0927 15:08:06.276521 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:06.276536 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:06.276646 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:06.276896 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:06.276899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:06.277110 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:06.277201 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:06.277207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:06.277920 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:06.277924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:06.278560 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:06.278569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:06.278972 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:06.278975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:06.279049 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:06.279518 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:06.279529 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:06.281274 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":86,"threads_started":1}
W20260927 15:08:06.281533 16427 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:06.276311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:43171 (local address 127.7.17.254:46207)
0927 15:08:06.276449 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:06.276454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:06.276604 (+ 150us) server_negotiation.cc:407] Connection header received
0927 15:08:06.276734 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:06.276738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:06.276783 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:06.276876 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:06.277318 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:06.277825 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:06.278668 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:06.278877 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:06.278968 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:06.281424 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.195:43171: BlockingRecv error: recv got EOF from 127.7.17.195:43171 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 15:08:06 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:06 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:06.927664 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:06.922201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.193:52203)
0927 15:08:06.922485 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:06.922503 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:06.922623 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:06.922922 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:06.922924 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:06.923105 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:06.923202 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:06.923207 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:06.924041 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:06.924044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:06.924616 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:06.924624 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:06.925154 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:06.925157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:06.925223 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:06.925707 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:06.925720 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:06.927541 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":84,"threads_started":1}
W20260927 15:08:06.927848 16431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:06.922340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:52203 (local address 127.7.17.254:46207)
0927 15:08:06.922649 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:06.922652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:06.922686 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:08:06.922725 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:06.922726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:06.922767 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:06.922851 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:06.923324 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:06.923921 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:06.924737 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:06.925041 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:06.925117 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:06.927712 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.193:52203: BlockingRecv error: recv got EOF from 127.7.17.193:52203 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":80,"threads_started":1}
Sep 27 15:08:07 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:07 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:07.103461 16432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:07.098034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.194:36101)
0927 15:08:07.098336 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:07.098350 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:07.098454 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:07.098719 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:07.098722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:07.098947 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:07.099044 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:07.099049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:07.099850 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:07.099853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:07.100417 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:07.100426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:07.100905 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:07.100907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:07.101009 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:07.101474 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:07.101485 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:07.103304 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":106,"threads_started":1}
W20260927 15:08:07.103559 16431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:07.098125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:36101 (local address 127.7.17.254:46207)
0927 15:08:07.098286 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:07.098292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:07.098417 (+ 125us) server_negotiation.cc:407] Connection header received
0927 15:08:07.098534 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:07.098537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:07.098581 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:07.098649 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:07.099189 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:07.099734 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:07.100544 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:07.100776 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:07.100900 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:07.103447 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.194:36101: BlockingRecv error: recv got EOF from 127.7.17.194:36101 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 15:08:07 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:07 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:07.288311 16433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:07.282461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.195:38339)
0927 15:08:07.282708 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:07.282722 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:07.282839 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:07.283137 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:07.283139 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:07.283362 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:07.283461 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:07.283476 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:07.284411 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:07.284415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:07.285219 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:07.285229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:07.285745 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:07.285749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:07.285894 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:07.286402 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:07.286414 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:07.288160 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
W20260927 15:08:07.288414 16431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:07.282548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:38339 (local address 127.7.17.254:46207)
0927 15:08:07.282716 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:07.282720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:07.282822 (+ 102us) server_negotiation.cc:407] Connection header received
0927 15:08:07.282929 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:07.282932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:07.282983 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:07.283060 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:07.283607 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:07.284273 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:07.285337 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:07.285629 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:07.285721 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:07.288302 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.195:38339: BlockingRecv error: recv got EOF from 127.7.17.195:38339 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 15:08:07 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:07 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:07.933740 16434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:07.928575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.193:51149)
0927 15:08:07.928871 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:07.928887 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:07.929005 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:07.929282 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:07.929285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:07.929488 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:07.929583 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:07.929589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:07.930344 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:07.930347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:07.930932 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:07.930939 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:07.931317 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:07.931319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:07.931385 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:07.931922 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:07.931937 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:07.933611 (+ 1674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":71,"threads_started":1}
W20260927 15:08:07.933923 16435 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:07.928638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:51149 (local address 127.7.17.254:46207)
0927 15:08:07.928944 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:07.928947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:07.928961 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:08:07.929099 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:07.929102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:07.929154 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:07.929248 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:07.929689 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:07.930253 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:07.931024 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:07.931239 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:07.931308 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:07.933751 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.193:51149: BlockingRecv error: recv got EOF from 127.7.17.193:51149 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
Sep 27 15:08:08 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:08 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:08.110199 16436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:08.104357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.194:56939)
0927 15:08:08.104621 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:08.104636 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:08.104734 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:08.105063 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:08.105067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:08.105269 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:08.105370 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:08.105378 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:08.106321 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:08.106324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:08.106915 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:08.106924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:08.107295 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:08.107297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:08.107357 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:08.107844 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:08.107901 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:08.110064 (+ 2163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
W20260927 15:08:08.110337 16435 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:08.104514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:56939 (local address 127.7.17.254:46207)
0927 15:08:08.104696 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:08.104700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:08.104719 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:08:08.104860 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:08.104863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:08.104916 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:08.105008 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:08.105503 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:08.106218 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:08.107007 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:08.107217 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:08.107286 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:08.110214 (+ 2928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.194:56939: BlockingRecv error: recv got EOF from 127.7.17.194:56939 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 27 15:08:08 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:08 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:08.294670 16437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:08.289116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.195:32947)
0927 15:08:08.289390 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:08.289404 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:08.289498 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:08.289723 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:08.289726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:08.289938 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:08.290034 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:08.290040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:08.290950 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:08.290952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:08.291549 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:08.291558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:08.292059 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:08.292062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:08.292128 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:08.292593 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:08.292606 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:08.294525 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":88,"threads_started":1}
W20260927 15:08:08.294839 16435 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:08.289253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:32947 (local address 127.7.17.254:46207)
0927 15:08:08.289391 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:08.289394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:08.289464 (+ 70us) server_negotiation.cc:407] Connection header received
0927 15:08:08.289570 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:08.289572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:08.289615 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:08.289684 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:08.290161 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:08.290743 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:08.291692 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:08.291934 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:08.292049 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:08.294679 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.195:32947: BlockingRecv error: recv got EOF from 127.7.17.195:32947 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 15:08:08 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:08 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:08.943461 16438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:08.934537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.193:41491)
0927 15:08:08.934790 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:08.934804 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:08.934894 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:08.935118 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:08.935120 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:08.935290 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:08.935384 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:08.935389 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:08.936250 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:08.936253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:08.936975 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:08.936989 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:08.937503 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:08.937507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:08.937587 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:08.941215 (+ 3628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:08.941230 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:08.943326 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
W20260927 15:08:08.943648 16439 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:08.934652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:41491 (local address 127.7.17.254:46207)
0927 15:08:08.934877 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:08.934881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:08.934894 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:08:08.934970 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:08.934972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:08.935014 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:08.935092 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:08.935505 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:08.936113 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:08.937090 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:08.937369 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:08.937481 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:08.943513 (+ 6032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.193:41491: BlockingRecv error: recv got EOF from 127.7.17.193:41491 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":67,"threads_started":1}
Sep 27 15:08:09 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:09 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:09.116528 16440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:09.111072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.194:42587)
0927 15:08:09.111329 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:09.111342 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:09.111471 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:09.111726 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:09.111729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:09.111903 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:09.112005 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:09.112012 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:09.112745 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:09.112747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:09.113355 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:09.113366 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:09.113878 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:09.113881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:09.113956 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:09.114570 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:09.114586 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:09.116388 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
W20260927 15:08:09.116669 16439 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:09.111208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:42587 (local address 127.7.17.254:46207)
0927 15:08:09.111347 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:09.111350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:09.111411 (+ 61us) server_negotiation.cc:407] Connection header received
0927 15:08:09.111557 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:09.111560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:09.111608 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:09.111694 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:09.112116 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:09.112647 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:09.113466 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:09.113769 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:09.113878 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:09.116547 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.194:42587: BlockingRecv error: recv got EOF from 127.7.17.194:42587 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 15:08:09 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:09 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:09.301272 16441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:09.295675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.195:55039)
0927 15:08:09.295960 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:09.295974 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:09.296075 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:09.296341 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:09.296343 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:09.296581 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:09.296680 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:09.296686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:09.297498 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:09.297501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:09.298073 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:09.298082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:09.298595 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:09.298598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:09.298666 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:09.299145 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:09.299157 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:09.301146 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":68,"threads_started":1}
W20260927 15:08:09.301463 16439 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:09.295737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:55039 (local address 127.7.17.254:46207)
0927 15:08:09.295875 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:09.295880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:09.296028 (+ 148us) server_negotiation.cc:407] Connection header received
0927 15:08:09.296155 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:09.296157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:09.296212 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:09.296271 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:09.296785 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:09.297367 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:09.298224 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:09.298475 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:09.298567 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:09.301333 (+ 2766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.195:55039: BlockingRecv error: recv got EOF from 127.7.17.195:55039 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 15:08:09 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:09 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:09.949574 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:09.944173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.193:53821)
0927 15:08:09.944440 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:09.944454 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:09.944571 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:09.944970 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:09.944973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:09.945141 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:09.945242 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:09.945247 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:09.945992 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:09.945995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:09.946571 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:09.946579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:09.947037 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:09.947040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:09.947098 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:09.947584 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:09.947594 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:09.949416 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
W20260927 15:08:09.949656 16443 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:09.944251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:53821 (local address 127.7.17.254:46207)
0927 15:08:09.944516 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:09.944521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:09.944534 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:08:09.944712 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:09.944716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:09.944772 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:09.944947 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:09.945337 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:09.945861 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:09.946686 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:09.946948 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:09.947028 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:09.949540 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.193:53821: BlockingRecv error: recv got EOF from 127.7.17.193:53821 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
Sep 27 15:08:10 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:10 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:10.122870 16444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:10.117410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.194:50641)
0927 15:08:10.117711 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:10.117725 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:10.117828 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:10.118070 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:10.118072 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:10.118247 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:10.118344 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:10.118349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:10.119103 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:10.119106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:10.119689 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:10.119699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:10.120195 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:10.120199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:10.120278 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:10.120799 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:10.120811 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:10.122696 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":102,"threads_started":1}
W20260927 15:08:10.123056 16443 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:10.117535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:50641 (local address 127.7.17.254:46207)
0927 15:08:10.117669 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:10.117673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:10.117790 (+ 117us) server_negotiation.cc:407] Connection header received
0927 15:08:10.117898 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:10.117901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:10.117945 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:10.118042 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:10.118433 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:10.119001 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:10.119785 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:10.120078 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:10.120174 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:10.122883 (+ 2709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.194:50641: BlockingRecv error: recv got EOF from 127.7.17.194:50641 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 15:08:10 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:10 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:10.307579 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:10.302218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.195:39297)
0927 15:08:10.302550 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:10.302565 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:10.302682 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:10.302929 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:10.302932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:10.303125 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:10.303215 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:10.303221 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:10.303944 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:10.303947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:10.304525 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:10.304533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:10.305010 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:10.305013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:10.305092 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:10.305580 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:10.305592 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:10.307431 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":101,"threads_started":1}
W20260927 15:08:10.307742 16443 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:10.302342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:39297 (local address 127.7.17.254:46207)
0927 15:08:10.302509 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:10.302514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:10.302638 (+ 124us) server_negotiation.cc:407] Connection header received
0927 15:08:10.302775 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:10.302778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:10.302826 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:10.302893 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:10.303327 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:10.303839 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:10.304635 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:10.304897 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:10.304996 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:10.307611 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.195:39297: BlockingRecv error: recv got EOF from 127.7.17.195:39297 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 15:08:10 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:10 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:10.956152 16446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:10.950548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.193:55939)
0927 15:08:10.950968 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:10.950990 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:10.951114 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:10.951397 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:10.951400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:10.951630 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:10.951752 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:10.951761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:10.952570 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:10.952573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:10.953166 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:10.953175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:10.953612 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:10.953616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:10.953691 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:10.954171 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:10.954183 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:10.955998 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":83,"threads_started":1}
W20260927 15:08:10.956264 16447 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:10.950672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:55939 (local address 127.7.17.254:46207)
0927 15:08:10.950880 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:10.950884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:10.951180 (+ 296us) server_negotiation.cc:407] Connection header received
0927 15:08:10.951227 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:10.951230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:10.951285 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:10.951372 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:10.951876 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:10.952459 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:10.953278 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:10.953514 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:10.953600 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:10.956150 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.193:55939: BlockingRecv error: recv got EOF from 127.7.17.193:55939 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":59,"threads_started":1}
Sep 27 15:08:11 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:11 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:11.129515 16448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:11.123749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.194:39219)
0927 15:08:11.124061 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:11.124077 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:11.124189 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:11.124529 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:11.124532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:11.124712 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:11.124868 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:11.124877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:11.125646 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:11.125648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:11.126250 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:11.126257 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:11.126662 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:11.126665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:11.126732 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:11.127263 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:11.127274 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:11.129365 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":138,"threads_started":1}
W20260927 15:08:11.129642 16447 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:11.123923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:39219 (local address 127.7.17.254:46207)
0927 15:08:11.124058 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:11.124062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:11.124145 (+ 83us) server_negotiation.cc:407] Connection header received
0927 15:08:11.124323 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:11.124327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:11.124378 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:11.124457 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:11.125028 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:11.125557 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:11.126345 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:11.126554 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:11.126626 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:11.129530 (+ 2904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.194:39219: BlockingRecv error: recv got EOF from 127.7.17.194:39219 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 15:08:11 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:11 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:11.313810 16449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:11.308354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.195:52685)
0927 15:08:11.308627 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:11.308641 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:11.308739 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:11.309056 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:11.309058 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:11.309240 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:11.309334 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:11.309373 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:11.310157 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:11.310160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:11.310755 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:11.310765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:11.311206 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:11.311209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:11.311277 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:11.311786 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:11.311799 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:11.313680 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":77,"threads_started":1}
W20260927 15:08:11.313966 16447 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:11.308435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:52685 (local address 127.7.17.254:46207)
0927 15:08:11.308572 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:11.308575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:11.308702 (+ 127us) server_negotiation.cc:407] Connection header received
0927 15:08:11.308890 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:11.308894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:11.308948 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:11.309024 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:11.309449 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:11.310034 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:11.310871 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:11.311097 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:11.311179 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:11.313820 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.195:52685: BlockingRecv error: recv got EOF from 127.7.17.195:52685 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 15:08:11 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:11 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:11.962606 16450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:11.957027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.193:47459)
0927 15:08:11.957299 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:11.957317 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:11.957435 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:11.957794 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:11.957796 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:11.957985 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:11.958082 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:11.958088 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:11.958873 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:11.958877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:11.959494 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:11.959503 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:11.960013 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:11.960017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:11.960106 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:11.960671 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:11.960684 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:11.962480 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":84,"threads_started":1}
W20260927 15:08:11.962744 16451 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:11.957145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:47459 (local address 127.7.17.254:46207)
0927 15:08:11.957506 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:11.957509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:11.957522 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:08:11.957562 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:11.957564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:11.957635 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:11.957724 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:11.958200 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:11.958755 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:11.959634 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:11.959861 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:11.960011 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:11.962614 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.193:47459: BlockingRecv error: recv got EOF from 127.7.17.193:47459 (error 108)
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":86,"threads_started":1}
Sep 27 15:08:12 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:12 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:12.135694 16452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:12.130259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.194:46873)
0927 15:08:12.130556 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:12.130570 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:12.130672 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:12.130886 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:12.130889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:12.131072 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:12.131168 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:12.131174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:12.131907 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:12.131910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:12.132507 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:12.132517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:12.133116 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:12.133119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:12.133187 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:12.133692 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:12.133704 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:12.135563 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":91,"threads_started":1}
W20260927 15:08:12.135898 16451 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:12.130355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:46873 (local address 127.7.17.254:46207)
0927 15:08:12.130516 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:12.130519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:12.130635 (+ 116us) server_negotiation.cc:407] Connection header received
0927 15:08:12.130746 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:12.130748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:12.130792 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:12.130862 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:12.131260 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:12.131783 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:12.132643 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:12.132967 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:12.133065 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:12.135728 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.194:46873: BlockingRecv error: recv got EOF from 127.7.17.194:46873 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 15:08:12 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:12 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.195@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:12.320363 16453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:12.314688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.195:35531)
0927 15:08:12.315001 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:12.315015 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:12.315133 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:12.315372 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:12.315374 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:12.315548 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:12.315638 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:12.315643 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:12.316418 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:12.316421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:12.317175 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:12.317187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:12.317805 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:12.317811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:12.317899 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:12.318375 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:12.318388 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:12.320237 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":81,"threads_started":1}
W20260927 15:08:12.320528 16451 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:12.314797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:35531 (local address 127.7.17.254:46207)
0927 15:08:12.314945 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:12.314949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:12.315117 (+ 168us) server_negotiation.cc:407] Connection header received
0927 15:08:12.315208 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:12.315210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:12.315260 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:12.315334 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:12.315722 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:12.316290 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:12.317277 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:12.317589 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:12.317697 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:12.320413 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.195:35531: BlockingRecv error: recv got EOF from 127.7.17.195:35531 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 15:08:12 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:12 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.193@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:12.969441 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:12.963600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.193:35069)
0927 15:08:12.963905 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:12.963921 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:12.964016 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:12.964323 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:12.964326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:12.964512 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:12.964623 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:12.964633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:12.965535 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:12.965538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:12.966132 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:12.966141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:12.966616 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:12.966620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:12.966695 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:12.967269 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:12.967281 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:12.969265 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":69,"threads_started":1}
W20260927 15:08:12.969549 16455 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:12.963709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:35069 (local address 127.7.17.254:46207)
0927 15:08:12.963999 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:12.964002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:12.964018 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:08:12.964107 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:12.964110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:12.964155 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:12.964228 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:12.964782 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:12.965431 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:12.966242 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:12.966483 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:12.966563 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:12.969418 (+ 2855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.193:35069: BlockingRecv error: recv got EOF from 127.7.17.193:35069 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":91,"threads_started":1}
Sep 27 15:08:13 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:08:13 dist-test-slave-hhpq krb5kdc[15396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.17.194@KRBTEST.COM for oryx/127.7.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:08:13.142004 16456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:13.136636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46207 (local address 127.7.17.194:43655)
0927 15:08:13.136933 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:13.136947 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:13.137054 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:13.137332 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:13.137334 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:13.137506 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:13.137601 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:13.137606 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:13.138338 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:13.138342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:13.138941 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:13.138950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:13.139412 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:13.139416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:13.139485 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:13.139973 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:13.139986 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:13.141873 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.17.254:46207: Server oryx/127.7.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":91,"threads_started":1}
W20260927 15:08:13.142181 16455 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:13.136659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:43655 (local address 127.7.17.254:46207)
0927 15:08:13.136801 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:13.136806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:13.137004 (+ 198us) server_negotiation.cc:407] Connection header received
0927 15:08:13.137138 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:13.137140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:13.137184 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:13.137250 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:13.137691 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:13.138228 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:13.139072 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:13.139297 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:13.139381 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:13.142055 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.17.194:43655: BlockingRecv error: recv got EOF from 127.7.17.194:43655 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:08:13.228878 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 15979
I20260927 15:08:13.233795 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 16130
I20260927 15:08:13.238674 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 16264
I20260927 15:08:13.243191 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 15909
2026-09-27T15:08:13Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17889 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T15:08:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:08:13Z Disabled control of system clock
I20260927 15:08:13.271493 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:42489
--webserver_interface=127.7.17.254
--webserver_port=0
--builtin_ntp_servers=127.7.17.212:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:42489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 15:08:13.386518 16460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:08:13.386819 16460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:08:13.386855 16460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:08:13.390996 16460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:08:13.391067 16460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:08:13.391081 16460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:08:13.391098 16460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:08:13.391111 16460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:08:13.396171 16460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:42489
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:42489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:08:13.397378 16460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:08:13.398813 16460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:08:13.407617 16466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:13.407714 16468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:13.407665 16465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:08:13.407881 16460 server_base.cc:1034] running on GCE node
I20260927 15:08:13.408557 16460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:08:13.409889 16460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:08:13.411082 16460 hybrid_clock.cc:648] HybridClock initialized: now 1790521693411052 us; error 43 us; skew 500 ppm
I20260927 15:08:13.413235 16460 webserver.cc:460] Webserver started at http://127.7.17.254:41315/ using document root <none> and password file <none>
I20260927 15:08:13.413877 16460 fs_manager.cc:359] Metadata directory not provided
I20260927 15:08:13.413949 16460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:08:13.414148 16460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:08:13.416393 16460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "14d2ecf14ab546e183acd048b1b5a3bd"
format_stamp: "Formatted at 2026-09-27 15:08:13 on dist-test-slave-hhpq"
server_key: "21f48c918717fa9ced5daee04f77965a"
server_key_iv: "ab8b87c18a96f75b67fd312b7c7d6330"
server_key_version: "encryptionkey@0"
I20260927 15:08:13.417073 16460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "14d2ecf14ab546e183acd048b1b5a3bd"
format_stamp: "Formatted at 2026-09-27 15:08:13 on dist-test-slave-hhpq"
server_key: "21f48c918717fa9ced5daee04f77965a"
server_key_iv: "ab8b87c18a96f75b67fd312b7c7d6330"
server_key_version: "encryptionkey@0"
I20260927 15:08:13.421362 16460 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:08:13.423959 16474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:13.425503 16460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:08:13.425642 16460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "14d2ecf14ab546e183acd048b1b5a3bd"
format_stamp: "Formatted at 2026-09-27 15:08:13 on dist-test-slave-hhpq"
server_key: "21f48c918717fa9ced5daee04f77965a"
server_key_iv: "ab8b87c18a96f75b67fd312b7c7d6330"
server_key_version: "encryptionkey@0"
I20260927 15:08:13.425763 16460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:08:13.456209 16460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:08:13.457058 16460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:08:13.457242 16460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:08:13.464066 16460 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:42489
I20260927 15:08:13.464088 16526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:42489 every 8 connection(s)
I20260927 15:08:13.465124 16460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:08:13.467751 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 16460
I20260927 15:08:13.467851 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:08:13.468140 7239 external_mini_cluster.cc:1428] Setting key 0bdea6bbad3dd0b6c77784ca655dbc70
I20260927 15:08:13.468717 16527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:08:13.475111 16527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d2ecf14ab546e183acd048b1b5a3bd: Bootstrap starting.
I20260927 15:08:13.477437 16527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14d2ecf14ab546e183acd048b1b5a3bd: Neither blocks nor log segments found. Creating new log.
I20260927 15:08:13.477445 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:13.469647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37760 (local address 127.7.17.254:42489)
0927 15:08:13.470302 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:13.470316 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:13.470353 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:08:13.471032 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:13.471049 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:13.471470 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:13.471786 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:13.472289 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:13.473306 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:13.474206 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:13.474692 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:13.474837 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:13.475669 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:08:13.475675 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:08:13.475688 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:08:13.476085 (+ 397us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:08:13.476140 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:08:13.476159 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:08:13.476243 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:13.476384 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 15:08:13.476646 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":138,"threads_started":1}
I20260927 15:08:13.478204 16527 log.cc:824] T 00000000000000000000000000000000 P 14d2ecf14ab546e183acd048b1b5a3bd: Log is configured to *not* fsync() on all Append() calls
I20260927 15:08:13.480118 16527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d2ecf14ab546e183acd048b1b5a3bd: No bootstrap required, opened a new log
I20260927 15:08:13.483603 16527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14d2ecf14ab546e183acd048b1b5a3bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d2ecf14ab546e183acd048b1b5a3bd" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42489 } }
I20260927 15:08:13.483840 16527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14d2ecf14ab546e183acd048b1b5a3bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:08:13.483896 16527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14d2ecf14ab546e183acd048b1b5a3bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d2ecf14ab546e183acd048b1b5a3bd, State: Initialized, Role: FOLLOWER
I20260927 15:08:13.484336 16527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14d2ecf14ab546e183acd048b1b5a3bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d2ecf14ab546e183acd048b1b5a3bd" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42489 } }
I20260927 15:08:13.484479 16527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14d2ecf14ab546e183acd048b1b5a3bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:08:13.484558 16527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14d2ecf14ab546e183acd048b1b5a3bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:08:13.484656 16527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14d2ecf14ab546e183acd048b1b5a3bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:08:13.485562 16527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14d2ecf14ab546e183acd048b1b5a3bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d2ecf14ab546e183acd048b1b5a3bd" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42489 } }
I20260927 15:08:13.485883 16527 leader_election.cc:302] T 00000000000000000000000000000000 P 14d2ecf14ab546e183acd048b1b5a3bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14d2ecf14ab546e183acd048b1b5a3bd; no voters:
I20260927 15:08:13.486168 16527 leader_election.cc:288] T 00000000000000000000000000000000 P 14d2ecf14ab546e183acd048b1b5a3bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:08:13.486284 16532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14d2ecf14ab546e183acd048b1b5a3bd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:08:13.486506 16532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14d2ecf14ab546e183acd048b1b5a3bd [term 1 LEADER]: Becoming Leader. State: Replica: 14d2ecf14ab546e183acd048b1b5a3bd, State: Running, Role: LEADER
I20260927 15:08:13.486821 16532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14d2ecf14ab546e183acd048b1b5a3bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d2ecf14ab546e183acd048b1b5a3bd" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42489 } }
I20260927 15:08:13.487309 16527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14d2ecf14ab546e183acd048b1b5a3bd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:08:13.488323 16533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d2ecf14ab546e183acd048b1b5a3bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14d2ecf14ab546e183acd048b1b5a3bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d2ecf14ab546e183acd048b1b5a3bd" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42489 } } }
I20260927 15:08:13.488458 16533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d2ecf14ab546e183acd048b1b5a3bd [sys.catalog]: This master's current role is: LEADER
I20260927 15:08:13.488862 16541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:08:13.488796 16534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d2ecf14ab546e183acd048b1b5a3bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14d2ecf14ab546e183acd048b1b5a3bd. Latest consensus state: current_term: 1 leader_uuid: "14d2ecf14ab546e183acd048b1b5a3bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d2ecf14ab546e183acd048b1b5a3bd" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42489 } } }
I20260927 15:08:13.488967 16534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d2ecf14ab546e183acd048b1b5a3bd [sys.catalog]: This master's current role is: LEADER
I20260927 15:08:13.491844 16541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:08:13.497752 16541 catalog_manager.cc:1347] Generated new cluster ID: 8baafe3ed99442d68bf5e897273a1510
I20260927 15:08:13.497941 16541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:08:13.521371 16541 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:08:13.522365 16541 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:08:13.528007 16541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14d2ecf14ab546e183acd048b1b5a3bd: Generated new TSK 0
I20260927 15:08:13.528699 16541 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:08:13.537601 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:42489
--builtin_ntp_servers=127.7.17.212:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 15:08:13.650815 16551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:08:13.651086 16551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:08:13.651121 16551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:08:13.655200 16551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:08:13.655267 16551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:08:13.655326 16551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:08:13.660209 16551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:42489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:08:13.661439 16551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:08:13.662856 16551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:08:13.672559 16557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:13.672597 16559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:08:13.672573 16551 server_base.cc:1034] running on GCE node
W20260927 15:08:13.672583 16556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:08:13.673310 16551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:08:13.673934 16551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:08:13.675101 16551 hybrid_clock.cc:648] HybridClock initialized: now 1790521693675070 us; error 43 us; skew 500 ppm
I20260927 15:08:13.676980 16551 webserver.cc:460] Webserver started at http://127.7.17.193:38331/ using document root <none> and password file <none>
I20260927 15:08:13.677556 16551 fs_manager.cc:359] Metadata directory not provided
I20260927 15:08:13.677618 16551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:08:13.677809 16551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:08:13.679876 16551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "2e2bcd9636a34335b38c05bf55abef4e"
format_stamp: "Formatted at 2026-09-27 15:08:13 on dist-test-slave-hhpq"
server_key: "e8dd0d8fd5db5c79a465806660c757b3"
server_key_iv: "dddb7fc1702988ec013081d17fba2ed9"
server_key_version: "encryptionkey@0"
I20260927 15:08:13.680423 16551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "2e2bcd9636a34335b38c05bf55abef4e"
format_stamp: "Formatted at 2026-09-27 15:08:13 on dist-test-slave-hhpq"
server_key: "e8dd0d8fd5db5c79a465806660c757b3"
server_key_iv: "dddb7fc1702988ec013081d17fba2ed9"
server_key_version: "encryptionkey@0"
I20260927 15:08:13.684157 16551 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 15:08:13.686806 16565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:13.688162 16551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:08:13.688268 16551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "2e2bcd9636a34335b38c05bf55abef4e"
format_stamp: "Formatted at 2026-09-27 15:08:13 on dist-test-slave-hhpq"
server_key: "e8dd0d8fd5db5c79a465806660c757b3"
server_key_iv: "dddb7fc1702988ec013081d17fba2ed9"
server_key_version: "encryptionkey@0"
I20260927 15:08:13.688391 16551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:08:13.700038 16551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:08:13.700736 16551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:08:13.700937 16551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:08:13.701515 16551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:08:13.702448 16551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:08:13.702494 16551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:13.702548 16551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:08:13.702574 16551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:13.710911 16551 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:43279
I20260927 15:08:13.710943 16678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:43279 every 8 connection(s)
I20260927 15:08:13.711808 16551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:08:13.713711 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 16551
I20260927 15:08:13.713810 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:08:13.714071 7239 external_mini_cluster.cc:1428] Setting key c2f727a5fff176538e4faa4c4aed7d99
I20260927 15:08:13.715867 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:42489
--builtin_ntp_servers=127.7.17.212:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 15:08:13.721098 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:13.713555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:38783 (local address 127.7.17.254:42489)
0927 15:08:13.713715 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:13.713720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:13.714925 (+ 1205us) server_negotiation.cc:407] Connection header received
0927 15:08:13.716289 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:13.716292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:13.716351 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:13.716433 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:13.717464 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:13.717988 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:13.719130 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:13.719348 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:13.719443 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:13.720597 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:08:13.720600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:08:13.720603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:08:13.720615 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:08:13.720635 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:08:13.720642 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:08:13.720700 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:13.720865 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 15:08:13.720938 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:08:13.722057 16681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:13.714055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:42489 (local address 127.7.17.193:38783)
0927 15:08:13.714762 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:13.714815 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:13.716082 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:13.716624 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:13.716633 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:13.716743 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:13.717324 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:13.717338 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:13.718114 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:13.718117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:13.719015 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:13.719024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:13.719503 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:13.719507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:13.719668 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:13.720324 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:08:13.720326 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:13.720372 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:08:13.720450 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:08:13.720710 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:08:13.720719 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:08:13.720725 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:08:13.720865 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 15:08:13.721237 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":110,"threads_started":1}
I20260927 15:08:13.723493 16679 heartbeater.cc:347] Connected to a master server at 127.7.17.254:42489
I20260927 15:08:13.723717 16679 heartbeater.cc:464] Registering TS with master...
I20260927 15:08:13.724268 16679 heartbeater.cc:511] Master 127.7.17.254:42489 requested a full tablet report, sending...
I20260927 15:08:13.725777 16486 ts_manager.cc:194] Registered new tserver with Master: 2e2bcd9636a34335b38c05bf55abef4e (127.7.17.193:43279)
I20260927 15:08:13.726907 16486 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.17.193:38783
W20260927 15:08:13.829278 16682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:08:13.829568 16682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:08:13.829620 16682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:08:13.833786 16682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:08:13.833843 16682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:08:13.833920 16682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:08:13.838692 16682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:42489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:08:13.839735 16682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:08:13.841143 16682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:08:13.850404 16690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:13.850410 16688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:08:13.850554 16682 server_base.cc:1034] running on GCE node
W20260927 15:08:13.850389 16687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:08:13.851046 16682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:08:13.851679 16682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:08:13.852865 16682 hybrid_clock.cc:648] HybridClock initialized: now 1790521693852848 us; error 31 us; skew 500 ppm
I20260927 15:08:13.854682 16682 webserver.cc:460] Webserver started at http://127.7.17.194:39449/ using document root <none> and password file <none>
I20260927 15:08:13.855239 16682 fs_manager.cc:359] Metadata directory not provided
I20260927 15:08:13.855305 16682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:08:13.855501 16682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:08:13.857611 16682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "9b98344d559c41e58a13dac6040a8fa8"
format_stamp: "Formatted at 2026-09-27 15:08:13 on dist-test-slave-hhpq"
server_key: "b3afe77c1835f6280e466c0277af742a"
server_key_iv: "bb62c34295b04b966c34a3b34efdf861"
server_key_version: "encryptionkey@0"
I20260927 15:08:13.858143 16682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "9b98344d559c41e58a13dac6040a8fa8"
format_stamp: "Formatted at 2026-09-27 15:08:13 on dist-test-slave-hhpq"
server_key: "b3afe77c1835f6280e466c0277af742a"
server_key_iv: "bb62c34295b04b966c34a3b34efdf861"
server_key_version: "encryptionkey@0"
I20260927 15:08:13.861868 16682 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:08:13.864439 16696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:13.865759 16682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:08:13.865912 16682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "9b98344d559c41e58a13dac6040a8fa8"
format_stamp: "Formatted at 2026-09-27 15:08:13 on dist-test-slave-hhpq"
server_key: "b3afe77c1835f6280e466c0277af742a"
server_key_iv: "bb62c34295b04b966c34a3b34efdf861"
server_key_version: "encryptionkey@0"
I20260927 15:08:13.866016 16682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:08:13.913439 16682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:08:13.914158 16682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:08:13.914331 16682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:08:13.914933 16682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:08:13.915899 16682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:08:13.915946 16682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:13.915999 16682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:08:13.916028 16682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:13.924432 16682 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:46853
I20260927 15:08:13.924456 16809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:46853 every 8 connection(s)
I20260927 15:08:13.925482 16682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:08:13.932528 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 16682
I20260927 15:08:13.932621 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:08:13.932924 7239 external_mini_cluster.cc:1428] Setting key 9985cd56321fdc02246c46285d855e00
I20260927 15:08:13.934954 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:42489
--builtin_ntp_servers=127.7.17.212:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 15:08:13.935170 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:13.927425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:42413 (local address 127.7.17.254:42489)
0927 15:08:13.927572 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:13.927576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:13.928763 (+ 1187us) server_negotiation.cc:407] Connection header received
0927 15:08:13.930259 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:13.930263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:13.930318 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:13.930403 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:13.931345 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:13.931877 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:13.933106 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:13.933325 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:13.933427 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:13.934698 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:08:13.934701 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:08:13.934702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:08:13.934714 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:08:13.934733 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:08:13.934742 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:08:13.934800 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:13.934965 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 15:08:13.935028 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:08:13.936128 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:13.927885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:42489 (local address 127.7.17.194:42413)
0927 15:08:13.928573 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:13.928619 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:13.930019 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:13.930568 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:13.930576 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:13.930670 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:13.931220 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:13.931234 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:13.931973 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:13.931975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:13.932984 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:13.932995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:13.933476 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:13.933480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:13.933681 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:13.934423 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:08:13.934424 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:13.934471 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:08:13.934544 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:08:13.934810 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:08:13.934819 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:08:13.934825 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:08:13.934955 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 15:08:13.935266 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":110,"threads_started":1}
I20260927 15:08:13.937352 16810 heartbeater.cc:347] Connected to a master server at 127.7.17.254:42489
I20260927 15:08:13.937610 16810 heartbeater.cc:464] Registering TS with master...
I20260927 15:08:13.938237 16810 heartbeater.cc:511] Master 127.7.17.254:42489 requested a full tablet report, sending...
I20260927 15:08:13.939504 16486 ts_manager.cc:194] Registered new tserver with Master: 9b98344d559c41e58a13dac6040a8fa8 (127.7.17.194:46853)
I20260927 15:08:13.940132 16486 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.17.194:42413
W20260927 15:08:14.045774 16813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:08:14.046072 16813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:08:14.046124 16813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:08:14.050305 16813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:08:14.050398 16813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:08:14.050486 16813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:08:14.055652 16813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:42489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:08:14.056743 16813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:08:14.058166 16813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:08:14.067700 16819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:14.067775 16818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:14.067819 16821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:08:14.068125 16813 server_base.cc:1034] running on GCE node
I20260927 15:08:14.068532 16813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:08:14.069262 16813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:08:14.070425 16813 hybrid_clock.cc:648] HybridClock initialized: now 1790521694070400 us; error 42 us; skew 500 ppm
I20260927 15:08:14.077991 16813 webserver.cc:460] Webserver started at http://127.7.17.195:39859/ using document root <none> and password file <none>
I20260927 15:08:14.078665 16813 fs_manager.cc:359] Metadata directory not provided
I20260927 15:08:14.078743 16813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:08:14.078949 16813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:08:14.081151 16813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "e4681af88b724627b343ae784426f12e"
format_stamp: "Formatted at 2026-09-27 15:08:14 on dist-test-slave-hhpq"
server_key: "ac91aec7bc069f61185bb99b66931816"
server_key_iv: "b7ae33c0fa0dd04837297b5b69261fb5"
server_key_version: "encryptionkey@0"
I20260927 15:08:14.081717 16813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "e4681af88b724627b343ae784426f12e"
format_stamp: "Formatted at 2026-09-27 15:08:14 on dist-test-slave-hhpq"
server_key: "ac91aec7bc069f61185bb99b66931816"
server_key_iv: "b7ae33c0fa0dd04837297b5b69261fb5"
server_key_version: "encryptionkey@0"
I20260927 15:08:14.085652 16813 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:08:14.088376 16827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:14.089984 16813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:08:14.090102 16813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "e4681af88b724627b343ae784426f12e"
format_stamp: "Formatted at 2026-09-27 15:08:14 on dist-test-slave-hhpq"
server_key: "ac91aec7bc069f61185bb99b66931816"
server_key_iv: "b7ae33c0fa0dd04837297b5b69261fb5"
server_key_version: "encryptionkey@0"
I20260927 15:08:14.090211 16813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:08:14.104238 16813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:08:14.105036 16813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:08:14.105199 16813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:08:14.105759 16813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:08:14.106700 16813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:08:14.106746 16813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:14.106784 16813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:08:14.106798 16813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:14.115200 16813 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:38763
I20260927 15:08:14.115231 16940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:38763 every 8 connection(s)
I20260927 15:08:14.116099 16813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:08:14.121457 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 16813
I20260927 15:08:14.121568 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:08:14.121868 7239 external_mini_cluster.cc:1428] Setting key 86bb84ed962cb54b327193b14cb9323c
I20260927 15:08:14.125485 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:14.117951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:35175 (local address 127.7.17.254:42489)
0927 15:08:14.118077 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:14.118080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:14.119155 (+ 1075us) server_negotiation.cc:407] Connection header received
0927 15:08:14.120558 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:14.120561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:14.120609 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:14.120683 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:14.121822 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:14.122362 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:14.123805 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:14.124036 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:14.124112 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:14.125121 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:08:14.125125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:08:14.125127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:08:14.125138 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:08:14.125158 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:08:14.125166 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:08:14.125221 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:14.125318 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 15:08:14.125368 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:08:14.126343 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:14.118354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:42489 (local address 127.7.17.195:35175)
0927 15:08:14.119021 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:14.119063 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:14.120311 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:14.120885 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:14.120893 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:14.120983 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:14.121674 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:14.121695 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:14.122539 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:14.122542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:14.123684 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:14.123693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:14.124156 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:14.124159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:14.124309 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:14.124846 (+ 537us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:08:14.124857 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:14.124915 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:08:14.124988 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:08:14.125220 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:08:14.125225 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:08:14.125228 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 15:08:14.125315 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 15:08:14.125545 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":112,"threads_started":1}
I20260927 15:08:14.127379 16941 heartbeater.cc:347] Connected to a master server at 127.7.17.254:42489
I20260927 15:08:14.127588 16941 heartbeater.cc:464] Registering TS with master...
I20260927 15:08:14.128069 16941 heartbeater.cc:511] Master 127.7.17.254:42489 requested a full tablet report, sending...
I20260927 15:08:14.129074 16486 ts_manager.cc:194] Registered new tserver with Master: e4681af88b724627b343ae784426f12e (127.7.17.195:38763)
I20260927 15:08:14.129637 16486 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.17.195:35175
I20260927 15:08:14.135802 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 15:08:14.138734 16530 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:14.137764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37770 (local address 127.7.17.254:42489)
0927 15:08:14.137944 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:14.137949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:14.138066 (+ 117us) server_negotiation.cc:407] Connection header received
0927 15:08:14.138189 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:14.138192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:14.138234 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:14.138298 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:14.138599 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37770: BlockingRecv error: recv got EOF from 127.0.0.1:37770 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:08:14.138957 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 16551
I20260927 15:08:14.143553 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 16682
I20260927 15:08:14.147969 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 16813
I20260927 15:08:14.152204 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 16460
2026-09-27T15:08:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (907 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T15:08:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:08:14Z Disabled control of system clock
I20260927 15:08:14.179013 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:39085
--webserver_interface=127.7.17.254
--webserver_port=0
--builtin_ntp_servers=127.7.17.212:38367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:39085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 15:08:14.289250 16952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:08:14.289526 16952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:08:14.289563 16952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:08:14.293588 16952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:08:14.293649 16952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:08:14.293663 16952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:08:14.293679 16952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:08:14.293694 16952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:08:14.298651 16952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:38367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:39085
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:39085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:08:14.299777 16952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:08:14.301263 16952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:08:14.309746 16958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:14.309755 16957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:08:14.309926 16952 server_base.cc:1034] running on GCE node
W20260927 15:08:14.309768 16960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:08:14.310675 16952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:08:14.311818 16952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:08:14.312973 16952 hybrid_clock.cc:648] HybridClock initialized: now 1790521694312951 us; error 35 us; skew 500 ppm
I20260927 15:08:14.314843 16952 webserver.cc:460] Webserver started at http://127.7.17.254:40917/ using document root <none> and password file <none>
I20260927 15:08:14.315426 16952 fs_manager.cc:359] Metadata directory not provided
I20260927 15:08:14.315493 16952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:08:14.315685 16952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:08:14.317813 16952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "a19442869eac4dc9a27d3e26f4be7a02"
format_stamp: "Formatted at 2026-09-27 15:08:14 on dist-test-slave-hhpq"
server_key: "c9ba5d02647b9267c12c2e9b0387b813"
server_key_iv: "82f439a8a942fa2422622e7ec55698c9"
server_key_version: "encryptionkey@0"
I20260927 15:08:14.318377 16952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "a19442869eac4dc9a27d3e26f4be7a02"
format_stamp: "Formatted at 2026-09-27 15:08:14 on dist-test-slave-hhpq"
server_key: "c9ba5d02647b9267c12c2e9b0387b813"
server_key_iv: "82f439a8a942fa2422622e7ec55698c9"
server_key_version: "encryptionkey@0"
I20260927 15:08:14.322264 16952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:08:14.325080 16966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:14.326393 16952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:08:14.326519 16952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "a19442869eac4dc9a27d3e26f4be7a02"
format_stamp: "Formatted at 2026-09-27 15:08:14 on dist-test-slave-hhpq"
server_key: "c9ba5d02647b9267c12c2e9b0387b813"
server_key_iv: "82f439a8a942fa2422622e7ec55698c9"
server_key_version: "encryptionkey@0"
I20260927 15:08:14.326627 16952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:08:14.331363 16952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:08:14.332029 16952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:08:14.332187 16952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:08:14.338924 16952 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:39085
I20260927 15:08:14.338939 17018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:39085 every 8 connection(s)
I20260927 15:08:14.340026 16952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:08:14.343381 17019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:08:14.344779 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 16952
I20260927 15:08:14.344916 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:08:14.345208 7239 external_mini_cluster.cc:1428] Setting key e39077284e51b84deb0604b129ad9239
I20260927 15:08:14.349939 17019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19442869eac4dc9a27d3e26f4be7a02: Bootstrap starting.
I20260927 15:08:14.350871 17022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:14.346771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48744 (local address 127.7.17.254:39085)
0927 15:08:14.347453 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:14.347469 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:14.347508 (+ 39us) server_negotiation.cc:407] Connection header received
0927 15:08:14.348160 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:14.348178 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:14.348534 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:14.348862 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:14.349604 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:08:14.349609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:08:14.349621 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:08:14.349629 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:08:14.349682 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:08:14.349696 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:08:14.349804 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:14.349898 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 15:08:14.350144 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":117,"threads_started":1}
I20260927 15:08:14.352262 17019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a19442869eac4dc9a27d3e26f4be7a02: Neither blocks nor log segments found. Creating new log.
I20260927 15:08:14.353114 17019 log.cc:824] T 00000000000000000000000000000000 P a19442869eac4dc9a27d3e26f4be7a02: Log is configured to *not* fsync() on all Append() calls
I20260927 15:08:14.354918 17019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19442869eac4dc9a27d3e26f4be7a02: No bootstrap required, opened a new log
I20260927 15:08:14.358450 17019 raft_consensus.cc:348] T 00000000000000000000000000000000 P a19442869eac4dc9a27d3e26f4be7a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19442869eac4dc9a27d3e26f4be7a02" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39085 } }
I20260927 15:08:14.358686 17019 raft_consensus.cc:374] T 00000000000000000000000000000000 P a19442869eac4dc9a27d3e26f4be7a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:08:14.358742 17019 raft_consensus.cc:729] T 00000000000000000000000000000000 P a19442869eac4dc9a27d3e26f4be7a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a19442869eac4dc9a27d3e26f4be7a02, State: Initialized, Role: FOLLOWER
I20260927 15:08:14.359150 17019 consensus_queue.cc:260] T 00000000000000000000000000000000 P a19442869eac4dc9a27d3e26f4be7a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19442869eac4dc9a27d3e26f4be7a02" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39085 } }
I20260927 15:08:14.359287 17019 raft_consensus.cc:388] T 00000000000000000000000000000000 P a19442869eac4dc9a27d3e26f4be7a02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:08:14.359349 17019 raft_consensus.cc:482] T 00000000000000000000000000000000 P a19442869eac4dc9a27d3e26f4be7a02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:08:14.359444 17019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a19442869eac4dc9a27d3e26f4be7a02 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:08:14.360337 17019 raft_consensus.cc:504] T 00000000000000000000000000000000 P a19442869eac4dc9a27d3e26f4be7a02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19442869eac4dc9a27d3e26f4be7a02" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39085 } }
I20260927 15:08:14.360666 17019 leader_election.cc:302] T 00000000000000000000000000000000 P a19442869eac4dc9a27d3e26f4be7a02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a19442869eac4dc9a27d3e26f4be7a02; no voters:
I20260927 15:08:14.360992 17019 leader_election.cc:288] T 00000000000000000000000000000000 P a19442869eac4dc9a27d3e26f4be7a02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:08:14.361102 17024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a19442869eac4dc9a27d3e26f4be7a02 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:08:14.361333 17024 raft_consensus.cc:686] T 00000000000000000000000000000000 P a19442869eac4dc9a27d3e26f4be7a02 [term 1 LEADER]: Becoming Leader. State: Replica: a19442869eac4dc9a27d3e26f4be7a02, State: Running, Role: LEADER
I20260927 15:08:14.361639 17024 consensus_queue.cc:237] T 00000000000000000000000000000000 P a19442869eac4dc9a27d3e26f4be7a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19442869eac4dc9a27d3e26f4be7a02" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39085 } }
I20260927 15:08:14.362263 17019 sys_catalog.cc:564] T 00000000000000000000000000000000 P a19442869eac4dc9a27d3e26f4be7a02 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:08:14.363237 17025 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19442869eac4dc9a27d3e26f4be7a02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a19442869eac4dc9a27d3e26f4be7a02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19442869eac4dc9a27d3e26f4be7a02" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39085 } } }
I20260927 15:08:14.363356 17025 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19442869eac4dc9a27d3e26f4be7a02 [sys.catalog]: This master's current role is: LEADER
I20260927 15:08:14.363701 17033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:08:14.363736 17026 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19442869eac4dc9a27d3e26f4be7a02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a19442869eac4dc9a27d3e26f4be7a02. Latest consensus state: current_term: 1 leader_uuid: "a19442869eac4dc9a27d3e26f4be7a02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19442869eac4dc9a27d3e26f4be7a02" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39085 } } }
I20260927 15:08:14.363940 17026 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19442869eac4dc9a27d3e26f4be7a02 [sys.catalog]: This master's current role is: LEADER
I20260927 15:08:14.367033 17033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:08:14.373435 17033 catalog_manager.cc:1347] Generated new cluster ID: 73dbeb96280b4264b7719b4ce99d3d0e
I20260927 15:08:14.373517 17033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:08:14.392868 17033 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:08:14.393261 17033 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:08:14.399667 17033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a19442869eac4dc9a27d3e26f4be7a02: Generated new TSK 0
I20260927 15:08:14.400542 17033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:08:14.411496 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39085
--builtin_ntp_servers=127.7.17.212:38367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 15:08:14.524653 17043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:08:14.525054 17043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:08:14.525110 17043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:08:14.530902 17043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:08:14.530987 17043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:08:14.531075 17043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:08:14.536109 17043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:38367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:08:14.537325 17043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:08:14.538729 17043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:08:14.548364 17048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:14.548422 17051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:14.548607 17049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:08:14.551946 17043 server_base.cc:1034] running on GCE node
I20260927 15:08:14.552425 17043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:08:14.553212 17043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:08:14.554400 17043 hybrid_clock.cc:648] HybridClock initialized: now 1790521694554369 us; error 61 us; skew 500 ppm
I20260927 15:08:14.556427 17043 webserver.cc:460] Webserver started at http://127.7.17.193:35817/ using document root <none> and password file <none>
I20260927 15:08:14.557109 17043 fs_manager.cc:359] Metadata directory not provided
I20260927 15:08:14.557188 17043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:08:14.557400 17043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:08:14.559614 17043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "60fb2cc259f341a9a378977eb83f3619"
format_stamp: "Formatted at 2026-09-27 15:08:14 on dist-test-slave-hhpq"
server_key: "a4cc5898258b7be8fca60e11922971b8"
server_key_iv: "5213e62e4842ae1edee8d22cc9045157"
server_key_version: "encryptionkey@0"
I20260927 15:08:14.560173 17043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "60fb2cc259f341a9a378977eb83f3619"
format_stamp: "Formatted at 2026-09-27 15:08:14 on dist-test-slave-hhpq"
server_key: "a4cc5898258b7be8fca60e11922971b8"
server_key_iv: "5213e62e4842ae1edee8d22cc9045157"
server_key_version: "encryptionkey@0"
I20260927 15:08:14.564113 17043 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:08:14.566823 17057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:14.568212 17043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:08:14.568352 17043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "60fb2cc259f341a9a378977eb83f3619"
format_stamp: "Formatted at 2026-09-27 15:08:14 on dist-test-slave-hhpq"
server_key: "a4cc5898258b7be8fca60e11922971b8"
server_key_iv: "5213e62e4842ae1edee8d22cc9045157"
server_key_version: "encryptionkey@0"
I20260927 15:08:14.568476 17043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:08:14.573347 17043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:08:14.573999 17043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:08:14.574162 17043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:08:14.574784 17043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:08:14.575904 17043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:08:14.575953 17043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:14.576006 17043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:08:14.576035 17043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:14.584506 17043 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:35211
I20260927 15:08:14.584537 17170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:35211 every 8 connection(s)
I20260927 15:08:14.585489 17043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:08:14.590036 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 17043
I20260927 15:08:14.590147 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:08:14.590444 7239 external_mini_cluster.cc:1428] Setting key 8ee672b20fa151c2d68c243bb8035b92
I20260927 15:08:14.591830 17022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:14.587383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:34583 (local address 127.7.17.254:39085)
0927 15:08:14.587527 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:14.587531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:14.588642 (+ 1111us) server_negotiation.cc:407] Connection header received
0927 15:08:14.590019 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:14.590022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:14.590072 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:14.590164 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:14.591396 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:08:14.591399 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:08:14.591402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:08:14.591412 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:08:14.591431 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:08:14.591439 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:08:14.591509 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:14.591663 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 15:08:14.591703 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:08:14.592381 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39085
--builtin_ntp_servers=127.7.17.212:38367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 15:08:14.592864 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:14.587800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:39085 (local address 127.7.17.193:34583)
0927 15:08:14.588490 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:14.588533 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:14.589778 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:14.590376 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:14.590386 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:14.590508 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:14.591104 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:08:14.591106 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:14.591163 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:08:14.591252 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:08:14.591514 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:08:14.591521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:08:14.591526 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:08:14.591715 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 15:08:14.591987 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":112,"threads_started":1}
I20260927 15:08:14.594143 17171 heartbeater.cc:347] Connected to a master server at 127.7.17.254:39085
I20260927 15:08:14.594383 17171 heartbeater.cc:464] Registering TS with master...
I20260927 15:08:14.594774 17171 heartbeater.cc:511] Master 127.7.17.254:39085 requested a full tablet report, sending...
I20260927 15:08:14.596258 16983 ts_manager.cc:194] Registered new tserver with Master: 60fb2cc259f341a9a378977eb83f3619 (127.7.17.193:35211)
W20260927 15:08:14.704008 17174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:08:14.704295 17174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:08:14.704334 17174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:08:14.708532 17174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:08:14.708593 17174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:08:14.708652 17174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:08:14.713523 17174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:38367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:08:14.714670 17174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:08:14.716591 17174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:08:14.728143 17180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:14.728161 17179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:14.728401 17182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:08:14.728629 17174 server_base.cc:1034] running on GCE node
I20260927 15:08:14.729166 17174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:08:14.729943 17174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:08:14.731122 17174 hybrid_clock.cc:648] HybridClock initialized: now 1790521694731084 us; error 46 us; skew 500 ppm
I20260927 15:08:14.732918 17174 webserver.cc:460] Webserver started at http://127.7.17.194:45607/ using document root <none> and password file <none>
I20260927 15:08:14.733461 17174 fs_manager.cc:359] Metadata directory not provided
I20260927 15:08:14.733518 17174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:08:14.733685 17174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:08:14.735855 17174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "699c70563f344d909220c536b6a921bc"
format_stamp: "Formatted at 2026-09-27 15:08:14 on dist-test-slave-hhpq"
server_key: "0882fa50a8d66f734425aaba7b1b7001"
server_key_iv: "2436e6266b2c90a41e05d6cd33583cef"
server_key_version: "encryptionkey@0"
I20260927 15:08:14.736378 17174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "699c70563f344d909220c536b6a921bc"
format_stamp: "Formatted at 2026-09-27 15:08:14 on dist-test-slave-hhpq"
server_key: "0882fa50a8d66f734425aaba7b1b7001"
server_key_iv: "2436e6266b2c90a41e05d6cd33583cef"
server_key_version: "encryptionkey@0"
I20260927 15:08:14.740127 17174 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:08:14.742880 17188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:14.744155 17174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:08:14.744269 17174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "699c70563f344d909220c536b6a921bc"
format_stamp: "Formatted at 2026-09-27 15:08:14 on dist-test-slave-hhpq"
server_key: "0882fa50a8d66f734425aaba7b1b7001"
server_key_iv: "2436e6266b2c90a41e05d6cd33583cef"
server_key_version: "encryptionkey@0"
I20260927 15:08:14.744418 17174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:08:14.749265 17174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:08:14.749876 17174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:08:14.750015 17174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:08:14.750528 17174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:08:14.751407 17174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:08:14.751452 17174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:14.751492 17174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:08:14.751514 17174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:14.760231 17174 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:37851
I20260927 15:08:14.760252 17301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:37851 every 8 connection(s)
I20260927 15:08:14.761246 17174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:08:14.767668 17022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:14.763185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:34695 (local address 127.7.17.254:39085)
0927 15:08:14.763374 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:14.763379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:14.764586 (+ 1207us) server_negotiation.cc:407] Connection header received
0927 15:08:14.765971 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:14.765974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:14.766015 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:14.766111 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:14.767285 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:08:14.767288 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:08:14.767290 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:08:14.767303 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:08:14.767325 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:08:14.767333 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:08:14.767416 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:14.767511 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 15:08:14.767550 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:08:14.768851 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:14.763678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:39085 (local address 127.7.17.194:34695)
0927 15:08:14.764385 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:14.764442 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:14.765707 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:14.766303 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:14.766310 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:14.766419 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:14.767010 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:08:14.767012 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:14.767056 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:08:14.767129 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:08:14.767405 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:08:14.767410 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:08:14.767414 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 15:08:14.767544 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 15:08:14.767780 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":133,"threads_started":1}
I20260927 15:08:14.769410 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 17174
I20260927 15:08:14.769526 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:08:14.769814 7239 external_mini_cluster.cc:1428] Setting key 22a8d07a82fc45596e0f809051315a2b
I20260927 15:08:14.770037 17302 heartbeater.cc:347] Connected to a master server at 127.7.17.254:39085
I20260927 15:08:14.770282 17302 heartbeater.cc:464] Registering TS with master...
I20260927 15:08:14.770720 17302 heartbeater.cc:511] Master 127.7.17.254:39085 requested a full tablet report, sending...
I20260927 15:08:14.771790 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39085
--builtin_ntp_servers=127.7.17.212:38367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 15:08:14.773170 16983 ts_manager.cc:194] Registered new tserver with Master: 699c70563f344d909220c536b6a921bc (127.7.17.194:37851)
W20260927 15:08:14.884584 17305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:08:14.884908 17305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:08:14.884949 17305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:08:14.889546 17305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:08:14.889612 17305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:08:14.889672 17305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:08:14.894611 17305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:38367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:08:14.895740 17305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:08:14.897222 17305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:08:14.907243 17310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:08:14.907488 17305 server_base.cc:1034] running on GCE node
W20260927 15:08:14.907260 17313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:14.907224 17311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:08:14.908021 17305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:08:14.908731 17305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:08:14.909936 17305 hybrid_clock.cc:648] HybridClock initialized: now 1790521694909910 us; error 40 us; skew 500 ppm
I20260927 15:08:14.911934 17305 webserver.cc:460] Webserver started at http://127.7.17.195:42961/ using document root <none> and password file <none>
I20260927 15:08:14.912545 17305 fs_manager.cc:359] Metadata directory not provided
I20260927 15:08:14.912616 17305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:08:14.912822 17305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:08:14.915100 17305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "29fecd4619ac45f3acfb1a819cb70a4d"
format_stamp: "Formatted at 2026-09-27 15:08:14 on dist-test-slave-hhpq"
server_key: "d963a1d462c3e600a1ab5c0d00bda3ee"
server_key_iv: "b94c0cc6e1bff998ec5d492527369c49"
server_key_version: "encryptionkey@0"
I20260927 15:08:14.915655 17305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "29fecd4619ac45f3acfb1a819cb70a4d"
format_stamp: "Formatted at 2026-09-27 15:08:14 on dist-test-slave-hhpq"
server_key: "d963a1d462c3e600a1ab5c0d00bda3ee"
server_key_iv: "b94c0cc6e1bff998ec5d492527369c49"
server_key_version: "encryptionkey@0"
I20260927 15:08:14.919612 17305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:08:14.922470 17319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:14.923795 17305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:08:14.923969 17305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "29fecd4619ac45f3acfb1a819cb70a4d"
format_stamp: "Formatted at 2026-09-27 15:08:14 on dist-test-slave-hhpq"
server_key: "d963a1d462c3e600a1ab5c0d00bda3ee"
server_key_iv: "b94c0cc6e1bff998ec5d492527369c49"
server_key_version: "encryptionkey@0"
I20260927 15:08:14.924085 17305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:08:14.928663 17305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:08:14.929316 17305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:08:14.929468 17305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:08:14.930006 17305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:08:14.930927 17305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:08:14.930974 17305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:14.931022 17305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:08:14.931048 17305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:14.939043 17305 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:37675
I20260927 15:08:14.939082 17432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:37675 every 8 connection(s)
I20260927 15:08:14.939950 17305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:08:14.946211 17022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:14.941835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:53629 (local address 127.7.17.254:39085)
0927 15:08:14.942032 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:14.942037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:14.943156 (+ 1119us) server_negotiation.cc:407] Connection header received
0927 15:08:14.944486 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:14.944490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:14.944530 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:14.944607 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:14.945789 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:08:14.945792 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:08:14.945795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:08:14.945808 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:08:14.945830 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:08:14.945837 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:08:14.945955 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:14.946056 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 15:08:14.946096 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 15:08:14.947121 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:14.942254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:39085 (local address 127.7.17.195:53629)
0927 15:08:14.942997 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:14.943040 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:14.944230 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:14.944801 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:14.944809 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:14.944975 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:14.945534 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:08:14.945536 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:14.945579 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:08:14.945651 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:08:14.945947 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:08:14.945952 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:08:14.945956 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 15:08:14.946096 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 15:08:14.946307 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":117,"threads_started":1}
I20260927 15:08:14.948199 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 17305
I20260927 15:08:14.948297 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:08:14.948289 17433 heartbeater.cc:347] Connected to a master server at 127.7.17.254:39085
I20260927 15:08:14.948529 17433 heartbeater.cc:464] Registering TS with master...
I20260927 15:08:14.948666 7239 external_mini_cluster.cc:1428] Setting key f3498bfe48e9cc2a8b8176272a9789c4
I20260927 15:08:14.949106 17433 heartbeater.cc:511] Master 127.7.17.254:39085 requested a full tablet report, sending...
I20260927 15:08:14.950173 16983 ts_manager.cc:194] Registered new tserver with Master: 29fecd4619ac45f3acfb1a819cb70a4d (127.7.17.195:37675)
I20260927 15:08:14.950822 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 15:08:14.953652 17022 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:08:14.952773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57388 (local address 127.7.17.254:39085)
0927 15:08:14.952934 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:14.952938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:14.953117 (+ 179us) server_negotiation.cc:407] Connection header received
0927 15:08:14.953234 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:14.953236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:14.953269 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:14.953327 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:14.953520 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57388: BlockingRecv error: recv got EOF from 127.0.0.1:57388 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 15:08:14.953908 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 17043
I20260927 15:08:14.959571 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 17174
I20260927 15:08:14.964450 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 17305
I20260927 15:08:14.969179 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 16952
2026-09-27T15:08:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (817 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:08:15 dist-test-slave-hhpq krb5kdc[17443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:08:15 dist-test-slave-hhpq krb5kdc[17443](info): set up 2 sockets
Sep 27 15:08:15 dist-test-slave-hhpq krb5kdc[17443](info): commencing operation
krb5kdc: starting...
W20260927 15:08:15.921828 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:08:16 dist-test-slave-hhpq krb5kdc[17443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:08:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:08:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:08:16.042028 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:33779
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:33779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:08:16.151758 17456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:08:16.152058 17456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:08:16.152100 17456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:08:16.156155 17456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:08:16.156224 17456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:08:16.156243 17456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:08:16.156260 17456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:08:16.156273 17456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:08:16.161700 17456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:33779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:33779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:08:16.162768 17456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:08:16.163751 17456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:08:16.172153 17464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:16.172153 17461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:16.172153 17462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:08:16.172191 17456 server_base.cc:1034] running on GCE node
I20260927 15:08:16.173027 17456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:08:16.174192 17456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:08:16.175422 17456 hybrid_clock.cc:648] HybridClock initialized: now 1790521696175399 us; error 33 us; skew 500 ppm
Sep 27 15:08:16 dist-test-slave-hhpq krb5kdc[17443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521696, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:08:16.178565 17456 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:08:16.179612 17456 webserver.cc:460] Webserver started at http://127.7.17.254:36519/ using document root <none> and password file <none>
I20260927 15:08:16.180172 17456 fs_manager.cc:359] Metadata directory not provided
I20260927 15:08:16.180241 17456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:08:16.180418 17456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:08:16.182731 17456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "fa37b185db7c4ff5bd35ffe60e649bde"
format_stamp: "Formatted at 2026-09-27 15:08:16 on dist-test-slave-hhpq"
server_key: "1e96e42cb74f4ecbe8d51650fe458204"
server_key_iv: "695fca8093e01f75866b77cc6c04348e"
server_key_version: "encryptionkey@0"
I20260927 15:08:16.183310 17456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "fa37b185db7c4ff5bd35ffe60e649bde"
format_stamp: "Formatted at 2026-09-27 15:08:16 on dist-test-slave-hhpq"
server_key: "1e96e42cb74f4ecbe8d51650fe458204"
server_key_iv: "695fca8093e01f75866b77cc6c04348e"
server_key_version: "encryptionkey@0"
I20260927 15:08:16.187438 17456 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260927 15:08:16.190112 17471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:16.191489 17456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:08:16.191614 17456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "fa37b185db7c4ff5bd35ffe60e649bde"
format_stamp: "Formatted at 2026-09-27 15:08:16 on dist-test-slave-hhpq"
server_key: "1e96e42cb74f4ecbe8d51650fe458204"
server_key_iv: "695fca8093e01f75866b77cc6c04348e"
server_key_version: "encryptionkey@0"
I20260927 15:08:16.191780 17456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:08:16.210505 17456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:08:16.213682 17456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:08:16.213855 17456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:08:16.220636 17456 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:33779
I20260927 15:08:16.220968 17523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:33779 every 8 connection(s)
I20260927 15:08:16.221798 17456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:08:16.224702 17524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:08:16.228175 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 17456
I20260927 15:08:16.228291 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:08:16.228549 7239 external_mini_cluster.cc:1428] Setting key 34bcce069d6564e1c2ff3c7ad46fa82e
I20260927 15:08:16.230657 17524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde: Bootstrap starting.
I20260927 15:08:16.232854 17524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde: Neither blocks nor log segments found. Creating new log.
I20260927 15:08:16.233649 17524 log.cc:824] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde: Log is configured to *not* fsync() on all Append() calls
I20260927 15:08:16.235455 17524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde: No bootstrap required, opened a new log
Sep 27 15:08:16 dist-test-slave-hhpq krb5kdc[17443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:08:16.238945 17524 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa37b185db7c4ff5bd35ffe60e649bde" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33779 } }
I20260927 15:08:16.239214 17524 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:08:16.239293 17524 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa37b185db7c4ff5bd35ffe60e649bde, State: Initialized, Role: FOLLOWER
I20260927 15:08:16.239856 17524 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa37b185db7c4ff5bd35ffe60e649bde" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33779 } }
I20260927 15:08:16.239990 17524 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:08:16.240051 17524 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:08:16.240145 17524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:08:16.241065 17524 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa37b185db7c4ff5bd35ffe60e649bde" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33779 } }
I20260927 15:08:16.241402 17524 leader_election.cc:302] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa37b185db7c4ff5bd35ffe60e649bde; no voters:
I20260927 15:08:16.241720 17524 leader_election.cc:288] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:08:16.241817 17529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:08:16.242022 17529 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde [term 1 LEADER]: Becoming Leader. State: Replica: fa37b185db7c4ff5bd35ffe60e649bde, State: Running, Role: LEADER
I20260927 15:08:16.242331 17529 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa37b185db7c4ff5bd35ffe60e649bde" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33779 } }
I20260927 15:08:16.243067 17524 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:08:16.243335 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:16.229891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58376 (local address 127.7.17.254:33779)
0927 15:08:16.230315 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:16.230334 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:16.230536 (+ 202us) server_negotiation.cc:407] Connection header received
0927 15:08:16.231276 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:16.231294 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:16.231645 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:16.231964 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:16.232717 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.233859 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.234683 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.235143 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.235284 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:16.237800 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:08:16.237805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:08:16.237820 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:08:16.237882 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:08:16.239979 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:16.240432 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:16.240437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:16.240441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:16.240550 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:16.240794 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:16.240797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:16.240799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:16.241256 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:08:16.241445 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:16.241759 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 15:08:16.242354 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":115,"threads_started":1}
I20260927 15:08:16.243912 17531 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa37b185db7c4ff5bd35ffe60e649bde. Latest consensus state: current_term: 1 leader_uuid: "fa37b185db7c4ff5bd35ffe60e649bde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa37b185db7c4ff5bd35ffe60e649bde" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33779 } } }
I20260927 15:08:16.243901 17530 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa37b185db7c4ff5bd35ffe60e649bde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa37b185db7c4ff5bd35ffe60e649bde" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 33779 } } }
I20260927 15:08:16.244033 17531 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde [sys.catalog]: This master's current role is: LEADER
I20260927 15:08:16.244033 17530 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde [sys.catalog]: This master's current role is: LEADER
I20260927 15:08:16.244383 17538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:08:16.247315 17538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:08:16.252521 17538 catalog_manager.cc:1347] Generated new cluster ID: 0f057e586fdb465d8718742bb13aa5a2
I20260927 15:08:16.252588 17538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:08:16.267572 17538 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:08:16.268638 17538 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:08:16.273005 17538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde: Generated new TSK 0
I20260927 15:08:16.273689 17538 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
I20260927 15:08:16.307802 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:33779
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:08:16.421527 17550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:08:16.421818 17550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:08:16.421870 17550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:08:16.426048 17550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:08:16.426117 17550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:08:16.426183 17550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:08:16.431592 17550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:33779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:08:16.432689 17550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:08:16.433768 17550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:08:16.443538 17558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:16.443542 17556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:16.443554 17555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:08:16.443890 17550 server_base.cc:1034] running on GCE node
I20260927 15:08:16.444494 17550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:08:16.445221 17550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:08:16.446424 17550 hybrid_clock.cc:648] HybridClock initialized: now 1790521696446397 us; error 35 us; skew 500 ppm
Sep 27 15:08:16 dist-test-slave-hhpq krb5kdc[17443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521696, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:08:16.449267 17550 init.cc:377] Logged in from keytab as kudu/127.7.17.193@KRBTEST.COM (short username kudu)
I20260927 15:08:16.450174 17550 webserver.cc:460] Webserver started at http://127.7.17.193:38327/ using document root <none> and password file <none>
I20260927 15:08:16.450726 17550 fs_manager.cc:359] Metadata directory not provided
I20260927 15:08:16.450786 17550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:08:16.451000 17550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:08:16.453087 17550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "2efc844705404b29ad65fe706e5ab4c2"
format_stamp: "Formatted at 2026-09-27 15:08:16 on dist-test-slave-hhpq"
server_key: "b3ccb7ddecc6d965c53e4aec7a3adcf6"
server_key_iv: "c8371e738e22cc6a4d0fc763e58d98c3"
server_key_version: "encryptionkey@0"
I20260927 15:08:16.453660 17550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "2efc844705404b29ad65fe706e5ab4c2"
format_stamp: "Formatted at 2026-09-27 15:08:16 on dist-test-slave-hhpq"
server_key: "b3ccb7ddecc6d965c53e4aec7a3adcf6"
server_key_iv: "c8371e738e22cc6a4d0fc763e58d98c3"
server_key_version: "encryptionkey@0"
I20260927 15:08:16.457688 17550 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:08:16.460168 17565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:16.461460 17550 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:08:16.461575 17550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "2efc844705404b29ad65fe706e5ab4c2"
format_stamp: "Formatted at 2026-09-27 15:08:16 on dist-test-slave-hhpq"
server_key: "b3ccb7ddecc6d965c53e4aec7a3adcf6"
server_key_iv: "c8371e738e22cc6a4d0fc763e58d98c3"
server_key_version: "encryptionkey@0"
I20260927 15:08:16.461675 17550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:08:16.471903 17550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:08:16.475006 17550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:08:16.475178 17550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:08:16.475755 17550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:08:16.476581 17550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:08:16.476627 17550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:16.476678 17550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:08:16.476703 17550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:16.485144 17550 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:36957
I20260927 15:08:16.485165 17678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:36957 every 8 connection(s)
I20260927 15:08:16.486029 17550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:08:16.493985 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 17550
I20260927 15:08:16.494092 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:08:16.494391 7239 external_mini_cluster.cc:1428] Setting key 99e69df7c6ecf34fef1460c65010f6dc
Sep 27 15:08:16 dist-test-slave-hhpq krb5kdc[17443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521696, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:08:16.499701 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:16.487751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:57181 (local address 127.7.17.254:33779)
0927 15:08:16.487869 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:16.487874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:16.488806 (+ 932us) server_negotiation.cc:407] Connection header received
0927 15:08:16.490127 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:16.490131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:16.490186 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:16.490262 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:16.491435 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.491975 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.493131 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.493362 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.493467 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:16.496117 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:08:16.496120 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:08:16.496122 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:08:16.496171 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:08:16.497698 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:16.498206 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:16.498209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:16.498211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:16.498313 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:16.498611 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:16.498614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:16.498616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:16.498845 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:08:16.498936 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:16.499397 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 15:08:16.499529 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 15:08:16.500524 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:16.488123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33779 (local address 127.7.17.193:57181)
0927 15:08:16.488572 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:16.488615 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:16.489912 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:16.490450 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:16.490457 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:16.490804 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:16.491286 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:16.491300 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.492094 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.492097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:16.493004 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:16.493019 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.493504 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.493508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:16.493676 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:16.494273 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:16.494294 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:16.495952 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:08:16.497815 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:08:16.497821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:08:16.497835 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:08:16.498114 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:08:16.498388 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:08:16.498390 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:08:16.498391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:08:16.498530 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:08:16.498930 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:08:16.498935 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:08:16.499182 (+ 247us) client_negotiation.cc:769] Sending connection context
0927 15:08:16.499335 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 15:08:16.499614 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":105,"threads_started":1}
I20260927 15:08:16.501736 17679 heartbeater.cc:347] Connected to a master server at 127.7.17.254:33779
I20260927 15:08:16.502002 17679 heartbeater.cc:464] Registering TS with master...
I20260927 15:08:16.502489 17679 heartbeater.cc:511] Master 127.7.17.254:33779 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.17.194@KRBTEST.COM; defaulting to no policy
I20260927 15:08:16.504004 17488 ts_manager.cc:194] Registered new tserver with Master: 2efc844705404b29ad65fe706e5ab4c2 (127.7.17.193:36957)
I20260927 15:08:16.505189 17488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:57181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
I20260927 15:08:16.521145 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:33779
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:08:16.631075 17684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:08:16.631378 17684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:08:16.631438 17684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:08:16.635567 17684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:08:16.635655 17684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:08:16.635738 17684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:08:16.641238 17684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:33779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:08:16.642347 17684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:08:16.643430 17684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:08:16.653061 17690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:16.653061 17689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:16.653124 17692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:08:16.653297 17684 server_base.cc:1034] running on GCE node
I20260927 15:08:16.653697 17684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:08:16.654290 17684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:08:16.655448 17684 hybrid_clock.cc:648] HybridClock initialized: now 1790521696655432 us; error 30 us; skew 500 ppm
Sep 27 15:08:16 dist-test-slave-hhpq krb5kdc[17443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521696, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:08:16.658105 17684 init.cc:377] Logged in from keytab as kudu/127.7.17.194@KRBTEST.COM (short username kudu)
I20260927 15:08:16.658977 17684 webserver.cc:460] Webserver started at http://127.7.17.194:41043/ using document root <none> and password file <none>
I20260927 15:08:16.659505 17684 fs_manager.cc:359] Metadata directory not provided
I20260927 15:08:16.659561 17684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:08:16.659725 17684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:08:16.661799 17684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "47c1874571ff4984bcf9a6680813dba6"
format_stamp: "Formatted at 2026-09-27 15:08:16 on dist-test-slave-hhpq"
server_key: "a3679c639163469f329e50d700036d43"
server_key_iv: "d0ba87b85923c776b7fa9a5585130f07"
server_key_version: "encryptionkey@0"
I20260927 15:08:16.662386 17684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "47c1874571ff4984bcf9a6680813dba6"
format_stamp: "Formatted at 2026-09-27 15:08:16 on dist-test-slave-hhpq"
server_key: "a3679c639163469f329e50d700036d43"
server_key_iv: "d0ba87b85923c776b7fa9a5585130f07"
server_key_version: "encryptionkey@0"
I20260927 15:08:16.666177 17684 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:08:16.668598 17699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:16.669912 17684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:08:16.670028 17684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "47c1874571ff4984bcf9a6680813dba6"
format_stamp: "Formatted at 2026-09-27 15:08:16 on dist-test-slave-hhpq"
server_key: "a3679c639163469f329e50d700036d43"
server_key_iv: "d0ba87b85923c776b7fa9a5585130f07"
server_key_version: "encryptionkey@0"
I20260927 15:08:16.670130 17684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:08:16.693815 17684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:08:16.697434 17684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:08:16.697628 17684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:08:16.698271 17684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:08:16.699131 17684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:08:16.699179 17684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:16.699232 17684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:08:16.699267 17684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:16.708022 17684 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:40149
I20260927 15:08:16.708039 17812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:40149 every 8 connection(s)
I20260927 15:08:16.709048 17684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:08:16.717551 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 17684
I20260927 15:08:16.717686 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:08:16.718039 7239 external_mini_cluster.cc:1428] Setting key 894db649bb496cb518b47afd2a294769
Sep 27 15:08:16 dist-test-slave-hhpq krb5kdc[17443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521696, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:08:16.723977 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:16.711190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:38341 (local address 127.7.17.254:33779)
0927 15:08:16.711329 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:16.711333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:16.712296 (+ 963us) server_negotiation.cc:407] Connection header received
0927 15:08:16.713681 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:16.713684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:16.713735 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:16.713822 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:16.715102 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.715648 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.716977 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.717207 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.717301 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:16.720209 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:08:16.720213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:08:16.720215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:08:16.720244 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:08:16.721726 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:16.722392 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:16.722395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:16.722397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:16.722483 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:16.722830 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:16.722833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:16.722835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:16.723039 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:08:16.723129 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:16.723666 (+ 537us) server_negotiation.cc:299] Negotiation successful
0927 15:08:16.723799 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:08:16.724862 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:16.711558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33779 (local address 127.7.17.194:38341)
0927 15:08:16.712112 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:16.712159 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:16.713442 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:16.714006 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:16.714014 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:16.714430 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:16.714929 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:16.714943 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.715792 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.715795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:16.716818 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:16.716861 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.717394 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.717399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:16.717604 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:16.718269 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:16.718290 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:16.720020 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:08:16.721855 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:08:16.721862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:08:16.721883 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:08:16.722270 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:08:16.722572 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:08:16.722576 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:08:16.722578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:08:16.722732 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:08:16.723145 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:08:16.723153 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:08:16.723424 (+ 271us) client_negotiation.cc:769] Sending connection context
0927 15:08:16.723604 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 15:08:16.723951 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":118,"threads_started":1}
I20260927 15:08:16.726075 17813 heartbeater.cc:347] Connected to a master server at 127.7.17.254:33779
I20260927 15:08:16.726317 17813 heartbeater.cc:464] Registering TS with master...
I20260927 15:08:16.726830 17813 heartbeater.cc:511] Master 127.7.17.254:33779 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.17.195@KRBTEST.COM; defaulting to no policy
I20260927 15:08:16.727912 17488 ts_manager.cc:194] Registered new tserver with Master: 47c1874571ff4984bcf9a6680813dba6 (127.7.17.194:40149)
I20260927 15:08:16.728519 17488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:38341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
I20260927 15:08:16.745939 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:33779
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:08:16.857851 17818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:08:16.858134 17818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:08:16.858179 17818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:08:16.862334 17818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:08:16.862408 17818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:08:16.862471 17818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:08:16.868177 17818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:33779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:08:16.869400 17818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:08:16.870462 17818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:08:16.880542 17826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:16.880542 17824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:08:16.880553 17823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:08:16.880954 17818 server_base.cc:1034] running on GCE node
I20260927 15:08:16.881418 17818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:08:16.882128 17818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:08:16.883285 17818 hybrid_clock.cc:648] HybridClock initialized: now 1790521696883254 us; error 46 us; skew 500 ppm
Sep 27 15:08:16 dist-test-slave-hhpq krb5kdc[17443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521696, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:08:16.886207 17818 init.cc:377] Logged in from keytab as kudu/127.7.17.195@KRBTEST.COM (short username kudu)
I20260927 15:08:16.887197 17818 webserver.cc:460] Webserver started at http://127.7.17.195:34233/ using document root <none> and password file <none>
I20260927 15:08:16.887758 17818 fs_manager.cc:359] Metadata directory not provided
I20260927 15:08:16.887821 17818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:08:16.888006 17818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:08:16.890209 17818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "940e84c124004aa3831c276638c85fb3"
format_stamp: "Formatted at 2026-09-27 15:08:16 on dist-test-slave-hhpq"
server_key: "d52ef9a7a1db50cbc813366c013c320d"
server_key_iv: "7b67c0a449bcd67f537a769752e94898"
server_key_version: "encryptionkey@0"
I20260927 15:08:16.890755 17818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "940e84c124004aa3831c276638c85fb3"
format_stamp: "Formatted at 2026-09-27 15:08:16 on dist-test-slave-hhpq"
server_key: "d52ef9a7a1db50cbc813366c013c320d"
server_key_iv: "7b67c0a449bcd67f537a769752e94898"
server_key_version: "encryptionkey@0"
I20260927 15:08:16.894793 17818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:08:16.897557 17833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:16.899087 17818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:08:16.899224 17818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "940e84c124004aa3831c276638c85fb3"
format_stamp: "Formatted at 2026-09-27 15:08:16 on dist-test-slave-hhpq"
server_key: "d52ef9a7a1db50cbc813366c013c320d"
server_key_iv: "7b67c0a449bcd67f537a769752e94898"
server_key_version: "encryptionkey@0"
I20260927 15:08:16.899360 17818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:08:16.929682 17818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:08:16.932719 17818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:08:16.932915 17818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:08:16.933526 17818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:08:16.934383 17818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:08:16.934438 17818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:16.934480 17818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:08:16.934509 17818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:08:16.942971 17818 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:41309
I20260927 15:08:16.942994 17946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:41309 every 8 connection(s)
I20260927 15:08:16.943984 17818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:08:16.952867 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 17818
I20260927 15:08:16.952993 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:08:16.953297 7239 external_mini_cluster.cc:1428] Setting key ff04d38d8bf17ae1e2391c462b161827
Sep 27 15:08:16 dist-test-slave-hhpq krb5kdc[17443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521696, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:08:16.957705 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:16.945896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:59763 (local address 127.7.17.254:33779)
0927 15:08:16.946022 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:16.946025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:16.946852 (+ 827us) server_negotiation.cc:407] Connection header received
0927 15:08:16.948191 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:16.948194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:16.948254 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:16.948333 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:16.949594 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.950115 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.951250 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.951470 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.951559 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:16.954243 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:08:16.954246 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:08:16.954250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:08:16.954281 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:08:16.955709 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:16.956233 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:16.956236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:16.956238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:16.956315 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:16.956667 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:16.956669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:16.956671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:16.956885 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:08:16.956964 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:16.957479 (+ 515us) server_negotiation.cc:299] Negotiation successful
0927 15:08:16.957584 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:08:16.958546 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:16.946227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:33779 (local address 127.7.17.195:59763)
0927 15:08:16.946682 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:16.946727 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:16.947968 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:16.948524 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:16.948531 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:16.948941 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:08:16.949446 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:16.949460 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.950238 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.950241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:16.951129 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:16.951138 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.951609 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.951613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:16.951777 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:16.952325 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:08:16.952341 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:08:16.954074 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:08:16.955868 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:08:16.955873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:08:16.955887 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:08:16.956150 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:08:16.956416 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:08:16.956419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:08:16.956420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:08:16.956581 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:08:16.956987 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:08:16.956992 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:08:16.957261 (+ 269us) client_negotiation.cc:769] Sending connection context
0927 15:08:16.957431 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 15:08:16.957706 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":112,"threads_started":1}
I20260927 15:08:16.959698 17947 heartbeater.cc:347] Connected to a master server at 127.7.17.254:33779
I20260927 15:08:16.959931 17947 heartbeater.cc:464] Registering TS with master...
I20260927 15:08:16.960469 17947 heartbeater.cc:511] Master 127.7.17.254:33779 requested a full tablet report, sending...
I20260927 15:08:16.961529 17488 ts_manager.cc:194] Registered new tserver with Master: 940e84c124004aa3831c276638c85fb3 (127.7.17.195:41309)
I20260927 15:08:16.962114 17488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:59763
I20260927 15:08:16.967314 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:08:16.976151 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:16.969143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58386 (local address 127.7.17.254:33779)
0927 15:08:16.969262 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:16.969267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:16.969609 (+ 342us) server_negotiation.cc:407] Connection header received
0927 15:08:16.969648 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:16.969650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:16.969690 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:16.969751 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:08:16.970445 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.970958 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.971775 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.971990 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.972076 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:16.973377 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:08:16.973380 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:08:16.973383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:08:16.973410 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:08:16.974575 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:16.974997 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:16.975001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:16.975002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:16.975075 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:08:16.975307 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:08:16.975310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:08:16.975312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:08:16.975536 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:08:16.975659 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:16.975886 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 15:08:16.975999 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:08:16.978935 17488 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:08:16.981945 17488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:08:16.996244 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:16.990075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:41309 (local address 127.0.0.1:50698)
0927 15:08:16.990643 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:16.990693 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:16.990817 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:16.991598 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:16.991603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:16.991626 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:08:16.991769 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:16.991775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.993225 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.993228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:16.994777 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:16.994784 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.995666 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.995669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:16.995783 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:16.995822 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 15:08:16.995928 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 15:08:16.996063 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":188,"thread_start_us":71,"threads_started":1}
I20260927 15:08:16.996228 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:16.990186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50698 (local address 127.7.17.195:41309)
0927 15:08:16.991106 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:16.991118 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:16.991140 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:08:16.991215 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:16.991220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:16.991435 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:16.991600 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:08:16.991877 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.993078 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.994903 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.995553 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.995653 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:16.995910 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:16.995965 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 15:08:16.996063 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":823,"thread_start_us":106,"threads_started":1}
I20260927 15:08:16.996555 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:16.990846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41372 (local address 127.7.17.194:40149)
0927 15:08:16.991263 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:16.991274 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:16.991594 (+ 320us) server_negotiation.cc:407] Connection header received
0927 15:08:16.991664 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:16.991669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:16.991860 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:16.991988 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:08:16.992300 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.993072 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.994771 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.995655 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.995759 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:16.995917 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:16.996319 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 15:08:16.996419 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":137,"threads_started":1}
I20260927 15:08:16.996797 17959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:16.990175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:36957 (local address 127.0.0.1:58478)
0927 15:08:16.991068 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:16.991080 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:16.991161 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:16.991963 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:16.991966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:16.991982 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:08:16.992077 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:16.992083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.993394 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.993397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:16.994745 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:16.994756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.995867 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.995870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:16.995959 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:16.995994 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 15:08:16.996052 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 15:08:16.996124 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"mutex_wait_us":8,"thread_start_us":113,"threads_started":1}
I20260927 15:08:16.996636 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:16.990728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58478 (local address 127.7.17.193:36957)
0927 15:08:16.991558 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:16.991572 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:16.991593 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:08:16.991660 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:16.991664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:16.991829 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:16.991982 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:08:16.992180 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.993282 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.994850 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.995655 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.995759 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:16.996365 (+ 606us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:16.996422 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 15:08:16.996508 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":59,"threads_started":1}
I20260927 15:08:16.997485 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:16.990710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:40149 (local address 127.0.0.1:41372)
0927 15:08:16.991394 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:16.991416 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:16.991503 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:16.992122 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:16.992125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:16.992140 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:08:16.992205 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:16.992210 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.993181 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.993184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:16.994630 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:16.994640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:16.995967 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:16.995971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:16.996068 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:16.996087 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 15:08:16.996139 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 15:08:16.996196 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":141,"thread_start_us":88,"threads_started":1}
I20260927 15:08:16.999729 17613 tablet_service.cc:1467] Processing CreateTablet for tablet 31d17946094749188619fab2b4011e4f (DEFAULT_TABLE table=test-table [id=b1c8cdc52c0c4b608827a265eedb9da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:08:16.999850 17881 tablet_service.cc:1467] Processing CreateTablet for tablet 31d17946094749188619fab2b4011e4f (DEFAULT_TABLE table=test-table [id=b1c8cdc52c0c4b608827a265eedb9da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:08:16.999907 17747 tablet_service.cc:1467] Processing CreateTablet for tablet 31d17946094749188619fab2b4011e4f (DEFAULT_TABLE table=test-table [id=b1c8cdc52c0c4b608827a265eedb9da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:08:17.000721 17613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31d17946094749188619fab2b4011e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:08:17.000907 17881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31d17946094749188619fab2b4011e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:08:17.000902 17747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31d17946094749188619fab2b4011e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:08:17.007737 17964 tablet_bootstrap.cc:492] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2: Bootstrap starting.
I20260927 15:08:17.009128 17966 tablet_bootstrap.cc:492] T 31d17946094749188619fab2b4011e4f P 47c1874571ff4984bcf9a6680813dba6: Bootstrap starting.
I20260927 15:08:17.009128 17965 tablet_bootstrap.cc:492] T 31d17946094749188619fab2b4011e4f P 940e84c124004aa3831c276638c85fb3: Bootstrap starting.
I20260927 15:08:17.010048 17964 tablet_bootstrap.cc:654] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2: Neither blocks nor log segments found. Creating new log.
I20260927 15:08:17.010910 17964 log.cc:824] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2: Log is configured to *not* fsync() on all Append() calls
I20260927 15:08:17.011305 17965 tablet_bootstrap.cc:654] T 31d17946094749188619fab2b4011e4f P 940e84c124004aa3831c276638c85fb3: Neither blocks nor log segments found. Creating new log.
I20260927 15:08:17.011765 17966 tablet_bootstrap.cc:654] T 31d17946094749188619fab2b4011e4f P 47c1874571ff4984bcf9a6680813dba6: Neither blocks nor log segments found. Creating new log.
I20260927 15:08:17.012566 17966 log.cc:824] T 31d17946094749188619fab2b4011e4f P 47c1874571ff4984bcf9a6680813dba6: Log is configured to *not* fsync() on all Append() calls
I20260927 15:08:17.012566 17965 log.cc:824] T 31d17946094749188619fab2b4011e4f P 940e84c124004aa3831c276638c85fb3: Log is configured to *not* fsync() on all Append() calls
I20260927 15:08:17.013320 17964 tablet_bootstrap.cc:492] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2: No bootstrap required, opened a new log
I20260927 15:08:17.013527 17964 ts_tablet_manager.cc:1397] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:08:17.014560 17965 tablet_bootstrap.cc:492] T 31d17946094749188619fab2b4011e4f P 940e84c124004aa3831c276638c85fb3: No bootstrap required, opened a new log
I20260927 15:08:17.014758 17965 ts_tablet_manager.cc:1397] T 31d17946094749188619fab2b4011e4f P 940e84c124004aa3831c276638c85fb3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:08:17.014882 17966 tablet_bootstrap.cc:492] T 31d17946094749188619fab2b4011e4f P 47c1874571ff4984bcf9a6680813dba6: No bootstrap required, opened a new log
I20260927 15:08:17.015053 17966 ts_tablet_manager.cc:1397] T 31d17946094749188619fab2b4011e4f P 47c1874571ff4984bcf9a6680813dba6: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260927 15:08:17.016994 17964 raft_consensus.cc:348] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "940e84c124004aa3831c276638c85fb3" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41309 } } peers { permanent_uuid: "2efc844705404b29ad65fe706e5ab4c2" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 36957 } } peers { permanent_uuid: "47c1874571ff4984bcf9a6680813dba6" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 40149 } }
I20260927 15:08:17.017222 17964 raft_consensus.cc:374] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:08:17.017273 17964 raft_consensus.cc:729] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efc844705404b29ad65fe706e5ab4c2, State: Initialized, Role: FOLLOWER
I20260927 15:08:17.017693 17964 consensus_queue.cc:260] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "940e84c124004aa3831c276638c85fb3" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41309 } } peers { permanent_uuid: "2efc844705404b29ad65fe706e5ab4c2" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 36957 } } peers { permanent_uuid: "47c1874571ff4984bcf9a6680813dba6" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 40149 } }
I20260927 15:08:17.018350 17679 heartbeater.cc:503] Master 127.7.17.254:33779 was elected leader, sending a full tablet report...
I20260927 15:08:17.018671 17964 ts_tablet_manager.cc:1428] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 15:08:17.018654 17965 raft_consensus.cc:348] T 31d17946094749188619fab2b4011e4f P 940e84c124004aa3831c276638c85fb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "940e84c124004aa3831c276638c85fb3" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41309 } } peers { permanent_uuid: "2efc844705404b29ad65fe706e5ab4c2" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 36957 } } peers { permanent_uuid: "47c1874571ff4984bcf9a6680813dba6" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 40149 } }
I20260927 15:08:17.018895 17965 raft_consensus.cc:374] T 31d17946094749188619fab2b4011e4f P 940e84c124004aa3831c276638c85fb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:08:17.018939 17965 raft_consensus.cc:729] T 31d17946094749188619fab2b4011e4f P 940e84c124004aa3831c276638c85fb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 940e84c124004aa3831c276638c85fb3, State: Initialized, Role: FOLLOWER
I20260927 15:08:17.019388 17965 consensus_queue.cc:260] T 31d17946094749188619fab2b4011e4f P 940e84c124004aa3831c276638c85fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "940e84c124004aa3831c276638c85fb3" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41309 } } peers { permanent_uuid: "2efc844705404b29ad65fe706e5ab4c2" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 36957 } } peers { permanent_uuid: "47c1874571ff4984bcf9a6680813dba6" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 40149 } }
I20260927 15:08:17.019832 17966 raft_consensus.cc:348] T 31d17946094749188619fab2b4011e4f P 47c1874571ff4984bcf9a6680813dba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "940e84c124004aa3831c276638c85fb3" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41309 } } peers { permanent_uuid: "2efc844705404b29ad65fe706e5ab4c2" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 36957 } } peers { permanent_uuid: "47c1874571ff4984bcf9a6680813dba6" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 40149 } }
I20260927 15:08:17.020092 17966 raft_consensus.cc:374] T 31d17946094749188619fab2b4011e4f P 47c1874571ff4984bcf9a6680813dba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:08:17.020141 17966 raft_consensus.cc:729] T 31d17946094749188619fab2b4011e4f P 47c1874571ff4984bcf9a6680813dba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47c1874571ff4984bcf9a6680813dba6, State: Initialized, Role: FOLLOWER
I20260927 15:08:17.020378 17965 ts_tablet_manager.cc:1428] T 31d17946094749188619fab2b4011e4f P 940e84c124004aa3831c276638c85fb3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:08:17.020555 17966 consensus_queue.cc:260] T 31d17946094749188619fab2b4011e4f P 47c1874571ff4984bcf9a6680813dba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "940e84c124004aa3831c276638c85fb3" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41309 } } peers { permanent_uuid: "2efc844705404b29ad65fe706e5ab4c2" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 36957 } } peers { permanent_uuid: "47c1874571ff4984bcf9a6680813dba6" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 40149 } }
I20260927 15:08:17.020727 17947 heartbeater.cc:503] Master 127.7.17.254:33779 was elected leader, sending a full tablet report...
I20260927 15:08:17.021167 17813 heartbeater.cc:503] Master 127.7.17.254:33779 was elected leader, sending a full tablet report...
I20260927 15:08:17.021492 17966 ts_tablet_manager.cc:1428] T 31d17946094749188619fab2b4011e4f P 47c1874571ff4984bcf9a6680813dba6: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260927 15:08:17.114259 17970 raft_consensus.cc:482] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:08:17.114477 17970 raft_consensus.cc:504] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "940e84c124004aa3831c276638c85fb3" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41309 } } peers { permanent_uuid: "2efc844705404b29ad65fe706e5ab4c2" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 36957 } } peers { permanent_uuid: "47c1874571ff4984bcf9a6680813dba6" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 40149 } }
I20260927 15:08:17.115679 17970 leader_election.cc:288] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 940e84c124004aa3831c276638c85fb3 (127.7.17.195:41309), 47c1874571ff4984bcf9a6680813dba6 (127.7.17.194:40149)
I20260927 15:08:17.119596 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:17.116092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:52339 (local address 127.7.17.194:40149)
0927 15:08:17.116215 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:17.116219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:17.116460 (+ 241us) server_negotiation.cc:407] Connection header received
0927 15:08:17.116582 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:17.116586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:17.116634 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:17.116695 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:08:17.116999 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:17.117524 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:17.118671 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:17.119052 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:17.119154 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:17.119206 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:17.119392 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 15:08:17.119469 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:08:17.119597 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:17.116075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:50299 (local address 127.7.17.195:41309)
0927 15:08:17.116211 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:17.116215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:17.116543 (+ 328us) server_negotiation.cc:407] Connection header received
0927 15:08:17.116658 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:17.116660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:17.116711 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:17.116796 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:08:17.117062 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:17.117549 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:17.118655 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:17.119048 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:17.119154 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:17.119206 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:17.119414 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 15:08:17.119481 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:08:17.119623 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:17.115962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:41309 (local address 127.7.17.193:50299)
0927 15:08:17.116428 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:17.116441 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:17.116552 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:17.116812 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:17.116814 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:17.116848 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:08:17.116961 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:17.116967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:17.117655 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:17.117658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:17.118538 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:17.118547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:17.119157 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:17.119160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:17.119303 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:17.119320 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 15:08:17.119386 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 15:08:17.119467 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":102,"thread_start_us":108,"threads_started":1}
I20260927 15:08:17.119619 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:17.115962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:40149 (local address 127.7.17.193:52339)
0927 15:08:17.116358 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:17.116370 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:08:17.116487 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:08:17.116733 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:08:17.116736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:08:17.116750 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:08:17.116898 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:17.116906 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:17.117634 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:17.117637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:17.118554 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:08:17.118563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:08:17.119157 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:08:17.119161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:08:17.119279 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:17.119301 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:08:17.119358 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 15:08:17.119456 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":66,"thread_start_us":144,"threads_started":1}
I20260927 15:08:17.120342 17901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d17946094749188619fab2b4011e4f" candidate_uuid: "2efc844705404b29ad65fe706e5ab4c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "940e84c124004aa3831c276638c85fb3" is_pre_election: true
I20260927 15:08:17.120337 17767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d17946094749188619fab2b4011e4f" candidate_uuid: "2efc844705404b29ad65fe706e5ab4c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47c1874571ff4984bcf9a6680813dba6" is_pre_election: true
I20260927 15:08:17.120620 17767 raft_consensus.cc:2454] T 31d17946094749188619fab2b4011e4f P 47c1874571ff4984bcf9a6680813dba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efc844705404b29ad65fe706e5ab4c2 in term 0.
I20260927 15:08:17.120625 17901 raft_consensus.cc:2454] T 31d17946094749188619fab2b4011e4f P 940e84c124004aa3831c276638c85fb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efc844705404b29ad65fe706e5ab4c2 in term 0.
I20260927 15:08:17.121065 17569 leader_election.cc:302] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efc844705404b29ad65fe706e5ab4c2, 47c1874571ff4984bcf9a6680813dba6; no voters:
I20260927 15:08:17.121299 17970 raft_consensus.cc:2781] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:08:17.121378 17970 raft_consensus.cc:482] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:08:17.121433 17970 raft_consensus.cc:3037] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:08:17.122406 17970 raft_consensus.cc:504] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "940e84c124004aa3831c276638c85fb3" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41309 } } peers { permanent_uuid: "2efc844705404b29ad65fe706e5ab4c2" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 36957 } } peers { permanent_uuid: "47c1874571ff4984bcf9a6680813dba6" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 40149 } }
I20260927 15:08:17.122715 17970 leader_election.cc:288] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2 [CANDIDATE]: Term 1 election: Requested vote from peers 940e84c124004aa3831c276638c85fb3 (127.7.17.195:41309), 47c1874571ff4984bcf9a6680813dba6 (127.7.17.194:40149)
I20260927 15:08:17.123076 17901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d17946094749188619fab2b4011e4f" candidate_uuid: "2efc844705404b29ad65fe706e5ab4c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "940e84c124004aa3831c276638c85fb3"
I20260927 15:08:17.123075 17767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d17946094749188619fab2b4011e4f" candidate_uuid: "2efc844705404b29ad65fe706e5ab4c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47c1874571ff4984bcf9a6680813dba6"
I20260927 15:08:17.123199 17901 raft_consensus.cc:3037] T 31d17946094749188619fab2b4011e4f P 940e84c124004aa3831c276638c85fb3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:08:17.123199 17767 raft_consensus.cc:3037] T 31d17946094749188619fab2b4011e4f P 47c1874571ff4984bcf9a6680813dba6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:08:17.124431 17767 raft_consensus.cc:2454] T 31d17946094749188619fab2b4011e4f P 47c1874571ff4984bcf9a6680813dba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efc844705404b29ad65fe706e5ab4c2 in term 1.
I20260927 15:08:17.124430 17901 raft_consensus.cc:2454] T 31d17946094749188619fab2b4011e4f P 940e84c124004aa3831c276638c85fb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efc844705404b29ad65fe706e5ab4c2 in term 1.
I20260927 15:08:17.124706 17568 leader_election.cc:302] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efc844705404b29ad65fe706e5ab4c2, 940e84c124004aa3831c276638c85fb3; no voters:
I20260927 15:08:17.124904 17970 raft_consensus.cc:2781] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:08:17.125104 17970 raft_consensus.cc:686] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2 [term 1 LEADER]: Becoming Leader. State: Replica: 2efc844705404b29ad65fe706e5ab4c2, State: Running, Role: LEADER
I20260927 15:08:17.125474 17970 consensus_queue.cc:237] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "940e84c124004aa3831c276638c85fb3" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41309 } } peers { permanent_uuid: "2efc844705404b29ad65fe706e5ab4c2" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 36957 } } peers { permanent_uuid: "47c1874571ff4984bcf9a6680813dba6" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 40149 } }
I20260927 15:08:17.128551 17487 catalog_manager.cc:5495] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efc844705404b29ad65fe706e5ab4c2 (127.7.17.193). New cstate: current_term: 1 leader_uuid: "2efc844705404b29ad65fe706e5ab4c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "940e84c124004aa3831c276638c85fb3" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efc844705404b29ad65fe706e5ab4c2" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 36957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47c1874571ff4984bcf9a6680813dba6" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 40149 } health_report { overall_health: UNKNOWN } } }
I20260927 15:08:17.149560 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:17.146123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58494 (local address 127.7.17.193:36957)
0927 15:08:17.146252 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 15:08:17.146255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:08:17.146269 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:08:17.146372 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:08:17.146374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:08:17.146429 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:08:17.146525 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:08:17.146713 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:17.147224 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:17.148112 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:08:17.148350 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:08:17.148434 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:08:17.148704 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:08:17.149198 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:08:17.149294 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 15:08:17.149358 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 15:08:17.149414 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:08:17.155514 17767 raft_consensus.cc:1261] T 31d17946094749188619fab2b4011e4f P 47c1874571ff4984bcf9a6680813dba6 [term 1 FOLLOWER]: Refusing update from remote peer 2efc844705404b29ad65fe706e5ab4c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:08:17.155514 17901 raft_consensus.cc:1261] T 31d17946094749188619fab2b4011e4f P 940e84c124004aa3831c276638c85fb3 [term 1 FOLLOWER]: Refusing update from remote peer 2efc844705404b29ad65fe706e5ab4c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:08:17.156252 17975 consensus_queue.cc:1035] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "940e84c124004aa3831c276638c85fb3" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:08:17.156447 17970 consensus_queue.cc:1035] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47c1874571ff4984bcf9a6680813dba6" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 40149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:08:17.172698 17486 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:08:17.172890 17486 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:08:17.175307 17486 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa37b185db7c4ff5bd35ffe60e649bde: Sending DeleteTablet for 3 replicas of tablet 31d17946094749188619fab2b4011e4f
I20260927 15:08:17.176159 17613 tablet_service.cc:1514] Processing DeleteTablet for tablet 31d17946094749188619fab2b4011e4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:08:17 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:58478
I20260927 15:08:17.176144 17881 tablet_service.cc:1514] Processing DeleteTablet for tablet 31d17946094749188619fab2b4011e4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:08:17 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:50698
I20260927 15:08:17.176497 17747 tablet_service.cc:1514] Processing DeleteTablet for tablet 31d17946094749188619fab2b4011e4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:08:17 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:41372
I20260927 15:08:17.177042 17990 tablet_replica.cc:330] T 31d17946094749188619fab2b4011e4f P 47c1874571ff4984bcf9a6680813dba6: stopping tablet replica
I20260927 15:08:17.177330 17989 tablet_replica.cc:330] T 31d17946094749188619fab2b4011e4f P 940e84c124004aa3831c276638c85fb3: stopping tablet replica
I20260927 15:08:17.177423 17990 raft_consensus.cc:2229] T 31d17946094749188619fab2b4011e4f P 47c1874571ff4984bcf9a6680813dba6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:08:17.177666 17988 tablet_replica.cc:330] T 31d17946094749188619fab2b4011e4f P 2efc844705404b29ad65fe706e5ab4c2: stopping tablet replica
I20260927 15:08:17.177826 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 17550
I20260927 15:08:17.177830 17990 raft_consensus.cc:2258] T 31d17946094749188619fab2b4011e4f P 47c1874571ff4984bcf9a6680813dba6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:08:17.178870 17989 raft_consensus.cc:2229] T 31d17946094749188619fab2b4011e4f P 940e84c124004aa3831c276638c85fb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:08:17.179337 17989 raft_consensus.cc:2258] T 31d17946094749188619fab2b4011e4f P 940e84c124004aa3831c276638c85fb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:08:17.180944 17989 ts_tablet_manager.cc:1905] T 31d17946094749188619fab2b4011e4f P 940e84c124004aa3831c276638c85fb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:08:17.181380 17990 ts_tablet_manager.cc:1905] T 31d17946094749188619fab2b4011e4f P 47c1874571ff4984bcf9a6680813dba6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:08:17.183760 17989 ts_tablet_manager.cc:1918] T 31d17946094749188619fab2b4011e4f P 940e84c124004aa3831c276638c85fb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:08:17.183877 17989 log.cc:1196] T 31d17946094749188619fab2b4011e4f P 940e84c124004aa3831c276638c85fb3: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/31d17946094749188619fab2b4011e4f
I20260927 15:08:17.183943 17990 ts_tablet_manager.cc:1918] T 31d17946094749188619fab2b4011e4f P 47c1874571ff4984bcf9a6680813dba6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:08:17.184029 17990 log.cc:1196] T 31d17946094749188619fab2b4011e4f P 47c1874571ff4984bcf9a6680813dba6: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521636012660-7239-0/minicluster-data/ts-1/wal/wals/31d17946094749188619fab2b4011e4f
W20260927 15:08:17.184237 17472 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.17.193:36957 (error 108)
I20260927 15:08:17.184316 17990 ts_tablet_manager.cc:1939] T 31d17946094749188619fab2b4011e4f P 47c1874571ff4984bcf9a6680813dba6: Deleting consensus metadata
I20260927 15:08:17.184392 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 17684
I20260927 15:08:17.185127 17989 ts_tablet_manager.cc:1939] T 31d17946094749188619fab2b4011e4f P 940e84c124004aa3831c276638c85fb3: Deleting consensus metadata
I20260927 15:08:17.185804 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:17.185459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:36957 (local address 127.0.0.1:58496)
0927 15:08:17.185645 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:17.185717 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.193:36957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260927 15:08:17.186079 17472 catalog_manager.cc:4584] TS 2efc844705404b29ad65fe706e5ab4c2 (127.7.17.193:36957): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31d17946094749188619fab2b4011e4f: Network error: Client connection negotiation failed: client connection to 127.7.17.193:36957: connect: Connection refused (error 111)
I20260927 15:08:17.186512 17474 catalog_manager.cc:4841] TS 940e84c124004aa3831c276638c85fb3 (127.7.17.195:41309): tablet 31d17946094749188619fab2b4011e4f (table test-table [id=b1c8cdc52c0c4b608827a265eedb9da5]) successfully deleted
I20260927 15:08:17.190371 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 17818
I20260927 15:08:17.190656 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:08:17.190390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:40149 (local address 127.0.0.1:41374)
0927 15:08:17.190525 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 15:08:17.190577 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.194:40149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 15:08:17.195981 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 17456
2026-09-27T15:08:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2227 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:08:17 dist-test-slave-hhpq krb5kdc[17994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:08:17 dist-test-slave-hhpq krb5kdc[17994](info): set up 2 sockets
Sep 27 15:08:17 dist-test-slave-hhpq krb5kdc[17994](info): commencing operation
krb5kdc: starting...
W20260927 15:08:18.145628 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:08:18 dist-test-slave-hhpq krb5kdc[17994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:08:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:08:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.17.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.17.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.17.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/rangeradmin_127.7.17.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.17.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/rangeradmin_127.7.17.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.17.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.17.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.17.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/rangerlookup_127.7.17.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.17.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/rangerlookup_127.7.17.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.17.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.17.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.17.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/HTTP_127.7.17.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.17.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/HTTP_127.7.17.212@KRBTEST.COM.keytab.
I20260927 15:08:18.311667 7239 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskgkjf1n/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/postgres -l logfile start
2026-09-27 15:08:20.818 UTC [18024] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 15:08:20.818 UTC [18024] LOG: listening on IPv4 address "127.7.17.212", port 38625
2026-09-27 15:08:20.822 UTC [18024] LOG: listening on Unix socket "/tmp/.s.PGSQL.38625"
2026-09-27 15:08:20.842 UTC [18027] LOG: database system was shut down at 2026-09-27 15:08:19 UTC
2026-09-27 15:08:20.845 UTC [18024] LOG: database system is ready to accept connections
I20260927 15:08:21.713292 7239 mini_postgres.cc:96] Postgres bound to 38625
2026-09-27 15:08:21.719 UTC [18035] FATAL: database "slave" does not exist
127.7.17.212:38625 - accepting connections
I20260927 15:08:21.720068 7239 mini_ranger.cc:161] Starting Ranger...
I20260927 15:08:21.734736 7239 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 15:08:22.237149 7239 mini_ranger.cc:87] Created ranger Postgres database
I20260927 15:08:22.237246 7239 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ranger-admin
2026-09-27 15:08:22,518 [I] DB FLAVOR :POSTGRES
2026-09-27 15:08:22,518 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:08:22,519 [I] Checking connection..
2026-09-27 15:08:22,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:38625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 15:08:22,758 [I] Checking connection passed.
2026-09-27 15:08:22,758 [I] --------- Verifying version history table ---------
2026-09-27 15:08:22,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:38625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:08:23,021 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:08:23,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:38625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:08:23,287 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:08:23,288 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 15:08:23,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:38625/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 15:08:23.552 UTC [18099] WARNING: there is no transaction in progress
2026-09-27 15:08:23,558 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 15:08:23,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:38625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:08:23,826 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 15:08:23,827 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:08:23,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:38625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 15:08:24,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:38625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 15:08:24,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:38625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hhpq.c.gcp-upstream.internal','N') ;"
2026-09-27 15:08:24,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:38625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:08:24,798 [I] Table x_portal_user does not exist in database ranger
2026-09-27 15:08:24,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:38625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:08:25,062 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 15:08:25,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:38625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:08:25,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:38625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:08:25,547 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 15:08:25,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:38625/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 15:08:26.562 UTC [18207] WARNING: there is no transaction in progress
2026-09-27 15:08:26.580 UTC [18207] WARNING: there is no transaction in progress
2026-09-27 15:08:26.598 UTC [18207] WARNING: there is no transaction in progress
2026-09-27 15:08:26.616 UTC [18207] WARNING: there is no transaction in progress
2026-09-27 15:08:26.635 UTC [18207] WARNING: there is no transaction in progress
2026-09-27 15:08:26.658 UTC [18207] WARNING: there is no transaction in progress
2026-09-27 15:08:26.724 UTC [18207] WARNING: there is no transaction in progress
2026-09-27 15:08:26.732 UTC [18207] WARNING: there is no transaction in progress
2026-09-27 15:08:26.748 UTC [18207] WARNING: there is no transaction in progress
2026-09-27 15:08:26.759 UTC [18207] WARNING: there is no transaction in progress
2026-09-27 15:08:26.771 UTC [18207] WARNING: there is no transaction in progress
2026-09-27 15:08:26.786 UTC [18207] WARNING: there is no transaction in progress
2026-09-27 15:08:26.797 UTC [18207] WARNING: there is no transaction in progress
2026-09-27 15:08:26.805 UTC [18207] WARNING: there is no transaction in progress
2026-09-27 15:08:26.812 UTC [18207] WARNING: there is no transaction in progress
2026-09-27 15:08:26.818 UTC [18207] WARNING: there is no transaction in progress
2026-09-27 15:08:27,524 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 15:08:27,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:38625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 15:08:27,764 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 15:08:27,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:38625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:08:28,026 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 15:08:28,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:38625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:08:28,290 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 15:08:28,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:38625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:08:28,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:38625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:08:28,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:38625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hhpq.c.gcp-upstream.internal';"
2026-09-27 15:08:29,045 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 15:08:29,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:38625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 15:08:29,305 [I] DB_PATCHES have already been applied
I20260927 15:08:29.310297 7239 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ranger-admin:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskgkjf1n/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 15:08:29.310360 7239 mini_ranger.cc:193] Using host: 127.7.17.212
I20260927 15:08:29.313551 7239 mini_ranger.cc:227] Ranger admin URL: http://127.7.17.212:40287
Sep 27, 2026 3:08:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:08:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:08:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 3:08:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 3:08:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 3:08:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 3:08:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 15:08:30.306650 7239 mini_ranger.cc:160] Time spent starting Ranger: real 8.587s user 0.003s sys 0.006s
Sep 27, 2026 3:08:51 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 3:08:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 3:08:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 3:08:51 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 3:08:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 3:08:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 3:08:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 3:08:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 3:08:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 3:08:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 3:08:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 3:08:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 3:08:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 3:08:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 3:08:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 3:08:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 3:08:51 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 3:08:52 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 15:08:52.613492 7239 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.17.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.17.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.17.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/rangerkms_127.7.17.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.17.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/rangerkms_127.7.17.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.17.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.17.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.17.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/HTTP_127.7.17.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.17.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/HTTP_127.7.17.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 15:08:52 dist-test-slave-hhpq krb5kdc[17994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521732, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 15:08:52.699460 7239 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 15:08:52.708592 7239 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 15:08:53.592691 7239 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 15:08:53.592775 7239 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ranger-kms
I20260927 15:08:53.592900 7239 mini_ranger_kms.cc:227] Using postgres at 127.7.17.212:38625
2026-09-27 15:08:53,768 [I] DB FLAVOR :POSTGRES
2026-09-27 15:08:53,768 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:08:53,768 [I] Checking connection
2026-09-27 15:08:53,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:38625/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 15:08:54,005 [I] connection success
2026-09-27 15:08:54,006 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 15:08:54,006 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 15:08:54,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:38625/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 15:08:54,246 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 15:08:54,246 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:08:54,246 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 15:08:54,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:38625/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 15:08:54,503 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 15:08:54.979672 7239 mini_ranger_kms.cc:310] Created kms service
I20260927 15:08:55.179643 7239 mini_ranger_kms.cc:326] Created kudu user
I20260927 15:08:55.222774 7239 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 15:08:55 dist-test-slave-hhpq krb5kdc[17994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521732, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.17.212@KRBTEST.COM
I20260927 15:08:55.586478 7239 mini_ranger_kms.cc:384] Added ranger policy
I20260927 15:08:55.586629 7239 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ranger-kms:/tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 15:08:55.586647 7239 mini_ranger_kms.cc:240] Using host: 127.7.17.212
I20260927 15:08:55.590591 7239 mini_ranger_kms.cc:276] Ranger KMS PID: 18387
I20260927 15:08:55.590714 7239 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.17.212:59907
Sep 27, 2026 3:08:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:08:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:08:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 3:08:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 3:08:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 3:08:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 15:08:57.588366 7239 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.889s user 0.019s sys 0.139s
I20260927 15:08:57.588493 7239 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 15:08:57.588609 7239 mini_ranger_kms.cc:401] 127.7.17.212:59907/kms/v1/keys
Sep 27 15:08:57 dist-test-slave-hhpq krb5kdc[17994](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790521737, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.17.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 15:08:58 dist-test-slave-hhpq krb5kdc[17994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521737, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.17.212@KRBTEST.COM for HTTP/127.7.17.212@KRBTEST.COM
Sep 27 15:08:58 dist-test-slave-hhpq krb5kdc[17994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521737, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.17.212@KRBTEST.COM for HTTP/127.7.17.212@KRBTEST.COM
Sep 27 15:09:00 dist-test-slave-hhpq krb5kdc[17994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:09:00.921264 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:35707
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:44521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:35707
--ranger_config_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.17.212:59907/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:09:01.057655 18438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:01.057943 18438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:01.057986 18438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:01.062150 18438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:09:01.062229 18438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:01.062248 18438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:01.062263 18438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:09:01.062278 18438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:09:01.068044 18438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:44521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.17.212:59907/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:35707
--ranger_config_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:35707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:09:01.069303 18438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:01.070420 18438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:01.080204 18443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:01.080283 18438 server_base.cc:1034] running on GCE node
W20260927 15:09:01.080467 18446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:01.080470 18444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:01.081280 18438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:01.082542 18438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:01.083750 18438 hybrid_clock.cc:648] HybridClock initialized: now 1790521741083718 us; error 47 us; skew 500 ppm
Sep 27 15:09:01 dist-test-slave-hhpq krb5kdc[17994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521741, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:01.086939 18438 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:09:01.088020 18438 webserver.cc:460] Webserver started at http://127.7.17.254:44911/ using document root <none> and password file <none>
I20260927 15:09:01.088639 18438 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:01.088702 18438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:01.088954 18438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:09:01 dist-test-slave-hhpq krb5kdc[17994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521741, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for HTTP/127.7.17.212@KRBTEST.COM
I20260927 15:09:01.142307 18438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "a1c24e56a4484031a3e653ebf2aa6ad1"
format_stamp: "Formatted at 2026-09-27 15:09:01 on dist-test-slave-hhpq"
server_key: "8e21187d5291db633cdd3db80c4e89a2"
server_key_iv: "91daa812b98ef6b35061054bcbf08f17"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:01.142932 18438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "a1c24e56a4484031a3e653ebf2aa6ad1"
format_stamp: "Formatted at 2026-09-27 15:09:01 on dist-test-slave-hhpq"
server_key: "8e21187d5291db633cdd3db80c4e89a2"
server_key_iv: "91daa812b98ef6b35061054bcbf08f17"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:01.147320 18438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:09:01.165318 18453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:01.167117 18438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:09:01.167282 18438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "a1c24e56a4484031a3e653ebf2aa6ad1"
format_stamp: "Formatted at 2026-09-27 15:09:01 on dist-test-slave-hhpq"
server_key: "8e21187d5291db633cdd3db80c4e89a2"
server_key_iv: "91daa812b98ef6b35061054bcbf08f17"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:01.167443 18438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:01.187834 18438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:01.191104 18438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:01.191283 18438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:01.198879 18438 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:35707
I20260927 15:09:01.198982 18505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:35707 every 8 connection(s)
I20260927 15:09:01.199891 18438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:09:01.202740 18506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:01.208509 18506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1: Bootstrap starting.
I20260927 15:09:01.210152 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 18438
I20260927 15:09:01.210295 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:09:01.210587 7239 external_mini_cluster.cc:1428] Setting key a40b325778bbf14916f717922664a388
I20260927 15:09:01.211313 18506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:01.212930 18506 log.cc:824] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:01.215288 18506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1: No bootstrap required, opened a new log
I20260927 15:09:01.219615 18506 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c24e56a4484031a3e653ebf2aa6ad1" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35707 } }
I20260927 15:09:01.219934 18506 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:01.220013 18506 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1c24e56a4484031a3e653ebf2aa6ad1, State: Initialized, Role: FOLLOWER
I20260927 15:09:01.220480 18506 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c24e56a4484031a3e653ebf2aa6ad1" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35707 } }
I20260927 15:09:01.220618 18506 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:09:01.220696 18506 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:09:01.220820 18506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 15:09:01 dist-test-slave-hhpq krb5kdc[17994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:09:01.222048 18506 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c24e56a4484031a3e653ebf2aa6ad1" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35707 } }
I20260927 15:09:01.222404 18506 leader_election.cc:302] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1c24e56a4484031a3e653ebf2aa6ad1; no voters:
I20260927 15:09:01.222769 18506 leader_election.cc:288] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:09:01.224092 18506 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:09:01.223228 18511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:09:01.224574 18511 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1 [term 1 LEADER]: Becoming Leader. State: Replica: a1c24e56a4484031a3e653ebf2aa6ad1, State: Running, Role: LEADER
I20260927 15:09:01.225567 18511 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c24e56a4484031a3e653ebf2aa6ad1" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35707 } }
I20260927 15:09:01.227609 18512 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1c24e56a4484031a3e653ebf2aa6ad1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c24e56a4484031a3e653ebf2aa6ad1" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35707 } } }
I20260927 15:09:01.227706 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:01.212031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.7.17.254:35707)
0927 15:09:01.212584 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:01.212603 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:01.212649 (+ 46us) server_negotiation.cc:407] Connection header received
0927 15:09:01.213542 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:01.213564 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:01.213980 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:01.214370 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:01.215287 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:01.216757 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:01.218152 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:01.218909 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:01.219078 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:01.222135 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:01.222142 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:01.222162 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:01.222199 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:01.224197 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:01.224750 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:01.224757 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:01.224764 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:01.224926 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:01.225262 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:01.225265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:01.225267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:01.225888 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:01.226065 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:01.226516 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 15:09:01.226794 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":388,"thread_start_us":194,"threads_started":1}
I20260927 15:09:01.227746 18512 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1 [sys.catalog]: This master's current role is: LEADER
I20260927 15:09:01.228279 18513 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1c24e56a4484031a3e653ebf2aa6ad1. Latest consensus state: current_term: 1 leader_uuid: "a1c24e56a4484031a3e653ebf2aa6ad1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c24e56a4484031a3e653ebf2aa6ad1" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35707 } } }
I20260927 15:09:01.228363 18513 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1 [sys.catalog]: This master's current role is: LEADER
I20260927 15:09:01.228530 18525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:09:01.228617 18506 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 15:09:01.231555 18525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:09:01.242417 18525 catalog_manager.cc:1347] Generated new cluster ID: 283dfc1f7cbb499fb9df3c01ea233a0b
I20260927 15:09:01.242579 18525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:09:01.257288 18525 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:09:01.258877 18525 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:09:01.277400 18525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1: Generated new TSK 0
I20260927 15:09:01.278594 18525 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
I20260927 15:09:01.980288 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.17.212:59907/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:35707
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:44521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:09:02.145613 18551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:02.146011 18551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:02.146080 18551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:02.152190 18551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:02.152277 18551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:02.152367 18551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:09:02.160275 18551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:44521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.17.212:59907/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:35707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:09:02.161893 18551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:02.163312 18551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:02.175997 18558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:02.175997 18559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:02.176262 18551 server_base.cc:1034] running on GCE node
W20260927 15:09:02.176477 18561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:02.176996 18551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:02.177728 18551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:02.178931 18551 hybrid_clock.cc:648] HybridClock initialized: now 1790521742178910 us; error 41 us; skew 500 ppm
Sep 27 15:09:02 dist-test-slave-hhpq krb5kdc[17994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521742, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:02.182251 18551 init.cc:377] Logged in from keytab as kudu/127.7.17.193@KRBTEST.COM (short username kudu)
I20260927 15:09:02.183477 18551 webserver.cc:460] Webserver started at http://127.7.17.193:40439/ using document root <none> and password file <none>
I20260927 15:09:02.184231 18551 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:02.184327 18551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:02.184569 18551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:09:02 dist-test-slave-hhpq krb5kdc[17994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521742, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for HTTP/127.7.17.212@KRBTEST.COM
I20260927 15:09:02.216945 18551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "63386367abbc4ea1bf457560cdbb96fe"
format_stamp: "Formatted at 2026-09-27 15:09:02 on dist-test-slave-hhpq"
server_key: "c3c6ab8c1f399a97d8b1c43ca4563e69"
server_key_iv: "8916e4fdbd1f76de555e7be2483638f5"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:02.217832 18551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "63386367abbc4ea1bf457560cdbb96fe"
format_stamp: "Formatted at 2026-09-27 15:09:02 on dist-test-slave-hhpq"
server_key: "c3c6ab8c1f399a97d8b1c43ca4563e69"
server_key_iv: "8916e4fdbd1f76de555e7be2483638f5"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:02.223438 18551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 15:09:02.257301 18568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:02.259420 18551 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.003s sys 0.000s
I20260927 15:09:02.259588 18551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "63386367abbc4ea1bf457560cdbb96fe"
format_stamp: "Formatted at 2026-09-27 15:09:02 on dist-test-slave-hhpq"
server_key: "c3c6ab8c1f399a97d8b1c43ca4563e69"
server_key_iv: "8916e4fdbd1f76de555e7be2483638f5"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:02.259744 18551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:02.291339 18551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:02.295413 18551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:02.295680 18551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:02.296440 18551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:09:02.297629 18551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:09:02.297742 18551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:02.297873 18551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:09:02.297953 18551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:02.308724 18551 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:46861
I20260927 15:09:02.308905 18683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:46861 every 8 connection(s)
I20260927 15:09:02.310220 18551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:09:02.311208 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 18551
I20260927 15:09:02.311339 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:09:02.311625 7239 external_mini_cluster.cc:1428] Setting key e9ec81a63513b0bdf29bee168e7c1443
Sep 27 15:09:02 dist-test-slave-hhpq krb5kdc[17994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521742, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:09:02.329459 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:02.314483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:54061 (local address 127.7.17.254:35707)
0927 15:09:02.316849 (+ 2366us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:02.316853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:02.316868 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:09:02.316913 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:02.316915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:02.316965 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:02.317046 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:02.318656 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:02.319523 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:02.321281 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:02.321644 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:02.321747 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:02.325011 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:02.325016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:02.325018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:02.325049 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:02.326879 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:02.327535 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:02.327539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:02.327541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:02.327631 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:02.328031 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:02.328034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:02.328036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:02.328289 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:02.328378 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:02.329123 (+ 745us) server_negotiation.cc:299] Negotiation successful
0927 15:09:02.329262 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2263,"thread_start_us":88,"threads_started":1}
I20260927 15:09:02.331516 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:02.313280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:35707 (local address 127.7.17.193:54061)
0927 15:09:02.315245 (+ 1965us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:02.315306 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:02.316391 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:02.317390 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:02.317400 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:02.317763 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:02.318476 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:02.318497 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:02.319672 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:02.319675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:02.321143 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:02.321155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:02.321944 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:02.321948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:02.322182 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:02.322935 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:02.322956 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:02.324743 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:02.327035 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:02.327042 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:02.327061 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:02.327421 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:02.327732 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:02.327735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:02.327737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:02.327931 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:02.328445 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:02.328451 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:02.328800 (+ 349us) client_negotiation.cc:769] Sending connection context
0927 15:09:02.329950 (+ 1150us) client_negotiation.cc:240] Negotiation successful
0927 15:09:02.330318 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1717,"thread_start_us":133,"threads_started":1}
I20260927 15:09:02.335090 18684 heartbeater.cc:347] Connected to a master server at 127.7.17.254:35707
I20260927 15:09:02.335412 18684 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:02.336025 18684 heartbeater.cc:511] Master 127.7.17.254:35707 requested a full tablet report, sending...
I20260927 15:09:02.337991 18466 ts_manager.cc:194] Registered new tserver with Master: 63386367abbc4ea1bf457560cdbb96fe (127.7.17.193:46861)
I20260927 15:09:02.339524 18466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:54061
WARNING: no policy specified for kudu/127.7.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
I20260927 15:09:02.369484 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.17.212:59907/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:35707
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:44521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:09:02 dist-test-slave-hhpq krb5kdc[17994](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790521742, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 15:09:02.535708 18690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:02.536145 18690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:02.536248 18690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:02.542083 18690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:02.542251 18690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:02.542424 18690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:09:02.550185 18690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:44521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.17.212:59907/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:35707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:09:02.551787 18690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:02.553218 18690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:02.566545 18697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:02.566545 18696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:02.567057 18699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:02.568356 18690 server_base.cc:1034] running on GCE node
I20260927 15:09:02.568923 18690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:02.569748 18690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:02.571035 18690 hybrid_clock.cc:648] HybridClock initialized: now 1790521742570970 us; error 85 us; skew 500 ppm
Sep 27 15:09:02 dist-test-slave-hhpq krb5kdc[17994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521742, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:02.574738 18690 init.cc:377] Logged in from keytab as kudu/127.7.17.194@KRBTEST.COM (short username kudu)
I20260927 15:09:02.575912 18690 webserver.cc:460] Webserver started at http://127.7.17.194:42267/ using document root <none> and password file <none>
I20260927 15:09:02.576696 18690 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:02.576874 18690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:02.577157 18690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:09:02 dist-test-slave-hhpq krb5kdc[17994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521742, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for HTTP/127.7.17.212@KRBTEST.COM
I20260927 15:09:02.596361 18690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "b44f42d7b14540ceb9791e13b173b14c"
format_stamp: "Formatted at 2026-09-27 15:09:02 on dist-test-slave-hhpq"
server_key: "64f70ef12f0dd913de3cf29f8da94f6b"
server_key_iv: "7369a22573c361dc01b9d1629d194364"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:02.597285 18690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "b44f42d7b14540ceb9791e13b173b14c"
format_stamp: "Formatted at 2026-09-27 15:09:02 on dist-test-slave-hhpq"
server_key: "64f70ef12f0dd913de3cf29f8da94f6b"
server_key_iv: "7369a22573c361dc01b9d1629d194364"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:02.606976 18690 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260927 15:09:02.629169 18706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:02.630877 18690 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:09:02.631009 18690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "b44f42d7b14540ceb9791e13b173b14c"
format_stamp: "Formatted at 2026-09-27 15:09:02 on dist-test-slave-hhpq"
server_key: "64f70ef12f0dd913de3cf29f8da94f6b"
server_key_iv: "7369a22573c361dc01b9d1629d194364"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:02.631138 18690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:02.641852 18690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:02.644727 18690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:02.644939 18690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:02.645568 18690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:09:02.646483 18690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:09:02.646531 18690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:02.646585 18690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:09:02.646638 18690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:02.657191 18690 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:38425
I20260927 15:09:02.657218 18819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:38425 every 8 connection(s)
I20260927 15:09:02.658473 18690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:09:02.659945 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 18690
I20260927 15:09:02.660059 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:09:02.660308 7239 external_mini_cluster.cc:1428] Setting key 4edd24db0527f339f416d8b5a7836541
WARNING: no policy specified for kudu/127.7.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.195@KRBTEST.COM" created.
Sep 27 15:09:02 dist-test-slave-hhpq krb5kdc[17994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521742, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:09:02.701901 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:02.685128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:60223 (local address 127.7.17.254:35707)
0927 15:09:02.685281 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:02.685287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:02.685893 (+ 606us) server_negotiation.cc:407] Connection header received
0927 15:09:02.687263 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:02.687268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:02.687333 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:02.687396 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:02.689365 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:02.690186 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:02.692177 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:02.692553 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:02.692642 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:02.696903 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:02.696908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:02.696911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:02.696946 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:02.698906 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:02.699744 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:02.699758 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:02.699760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:02.699871 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:02.700379 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:02.700384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:02.700386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:02.700669 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:02.700774 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:02.701564 (+ 790us) server_negotiation.cc:299] Negotiation successful
0927 15:09:02.701713 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:09:02.702755 18824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:02.683713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:35707 (local address 127.7.17.194:60223)
0927 15:09:02.685668 (+ 1955us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:02.685743 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:02.686964 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:02.687918 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:02.687929 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:02.688345 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:02.689161 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:02.689184 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:02.690446 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:02.690450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:02.691998 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:02.692011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:02.692923 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:02.692928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:02.693160 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:02.693962 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:02.693986 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:02.696591 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:02.699055 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:02.699063 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:02.699085 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:02.699594 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:02.699996 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:02.700000 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:02.700002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:02.700240 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:02.700805 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:02.700814 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:02.701262 (+ 448us) client_negotiation.cc:769] Sending connection context
0927 15:09:02.701459 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 15:09:02.701780 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1683,"thread_start_us":181,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
I20260927 15:09:02.704011 18820 heartbeater.cc:347] Connected to a master server at 127.7.17.254:35707
I20260927 15:09:02.704324 18820 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:02.704373 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.17.212:59907/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:35707
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:44521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:09:02.704968 18820 heartbeater.cc:511] Master 127.7.17.254:35707 requested a full tablet report, sending...
I20260927 15:09:02.706133 18466 ts_manager.cc:194] Registered new tserver with Master: b44f42d7b14540ceb9791e13b173b14c (127.7.17.194:38425)
I20260927 15:09:02.706696 18466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:60223
W20260927 15:09:02.866088 18825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:02.866601 18825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:02.866722 18825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:02.873061 18825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:02.873234 18825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:02.873483 18825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:09:02.880647 18825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:44521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.17.212:59907/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:35707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:09:02.882201 18825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:02.883317 18825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 27 15:09:02 dist-test-slave-hhpq krb5kdc[17994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521742, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for HTTP/127.7.17.212@KRBTEST.COM
W20260927 15:09:02.905550 18830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:02.906267 18833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:02.905560 18831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:02.906821 18825 server_base.cc:1034] running on GCE node
I20260927 15:09:02.907416 18825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:02.908386 18825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:02.911670 18825 hybrid_clock.cc:648] HybridClock initialized: now 1790521742911573 us; error 101 us; skew 500 ppm
Sep 27 15:09:02 dist-test-slave-hhpq krb5kdc[17994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521742, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:02.915773 18825 init.cc:377] Logged in from keytab as kudu/127.7.17.195@KRBTEST.COM (short username kudu)
I20260927 15:09:02.917158 18825 webserver.cc:460] Webserver started at http://127.7.17.195:43859/ using document root <none> and password file <none>
I20260927 15:09:02.918012 18825 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:02.918177 18825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:02.918473 18825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:09:02 dist-test-slave-hhpq krb5kdc[17994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521742, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for HTTP/127.7.17.212@KRBTEST.COM
I20260927 15:09:02.934844 18825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "d0dbb6edef6a4f4dbe8377edff7bde77"
format_stamp: "Formatted at 2026-09-27 15:09:02 on dist-test-slave-hhpq"
server_key: "ee4057672c02fabf8e0753b82455e3d3"
server_key_iv: "76e1a3056746dc4347e05a7d0b86b44e"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:02.935679 18825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "d0dbb6edef6a4f4dbe8377edff7bde77"
format_stamp: "Formatted at 2026-09-27 15:09:02 on dist-test-slave-hhpq"
server_key: "ee4057672c02fabf8e0753b82455e3d3"
server_key_iv: "76e1a3056746dc4347e05a7d0b86b44e"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:02.946662 18825 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.005s sys 0.000s
I20260927 15:09:02.981643 18840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:02.983763 18825 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260927 15:09:02.983946 18825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "d0dbb6edef6a4f4dbe8377edff7bde77"
format_stamp: "Formatted at 2026-09-27 15:09:02 on dist-test-slave-hhpq"
server_key: "ee4057672c02fabf8e0753b82455e3d3"
server_key_iv: "76e1a3056746dc4347e05a7d0b86b44e"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:02.984117 18825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:03.019112 18825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:03.023180 18825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:03.023460 18825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:03.024272 18825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:09:03.025534 18825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:09:03.025645 18825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:03.025739 18825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:09:03.025797 18825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:03.039398 18825 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:33631
I20260927 15:09:03.042140 18825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:09:03.040939 18953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:33631 every 8 connection(s)
I20260927 15:09:03.049968 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 18825
I20260927 15:09:03.050097 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:09:03.050369 7239 external_mini_cluster.cc:1428] Setting key c46a7d4d0628d095a42d79920e7fc9f9
Sep 27 15:09:03 dist-test-slave-hhpq krb5kdc[17994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521742, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:09:03.064337 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:03.048336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:57537 (local address 127.7.17.254:35707)
0927 15:09:03.048469 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:03.048473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:03.049006 (+ 533us) server_negotiation.cc:407] Connection header received
0927 15:09:03.051614 (+ 2608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:03.051618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:03.051681 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:03.051781 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:03.053589 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.054429 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.056221 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.056583 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.056670 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:03.059888 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:03.059893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:03.059896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:03.059926 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:03.061703 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:03.062394 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:03.062398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:03.062400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:03.062492 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:03.062901 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:03.062904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:03.062906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:03.063140 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:03.063235 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:03.064025 (+ 790us) server_negotiation.cc:299] Negotiation successful
0927 15:09:03.064167 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:09:03.066604 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:03.047150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:35707 (local address 127.7.17.195:57537)
0927 15:09:03.048800 (+ 1650us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:03.048894 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:03.051303 (+ 2409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:03.052152 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:03.052164 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:03.052578 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:03.053398 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:03.053419 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.054602 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.054605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:03.056050 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:03.056064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.056924 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.056928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:03.057140 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:03.057858 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:03.057882 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:03.059666 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:03.061862 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:03.061869 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:03.061890 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:03.062257 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:03.062594 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:03.062598 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:03.062600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:03.062792 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:03.063310 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:03.063317 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:03.063698 (+ 381us) client_negotiation.cc:769] Sending connection context
0927 15:09:03.064871 (+ 1173us) client_negotiation.cc:240] Negotiation successful
0927 15:09:03.065259 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1419,"thread_start_us":170,"threads_started":1}
I20260927 15:09:03.068260 18954 heartbeater.cc:347] Connected to a master server at 127.7.17.254:35707
I20260927 15:09:03.068598 18954 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:03.069243 18954 heartbeater.cc:511] Master 127.7.17.254:35707 requested a full tablet report, sending...
I20260927 15:09:03.070523 18466 ts_manager.cc:194] Registered new tserver with Master: d0dbb6edef6a4f4dbe8377edff7bde77 (127.7.17.195:33631)
I20260927 15:09:03.071254 18466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:57537
I20260927 15:09:03.076050 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:09:03.096117 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:03.085377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56766 (local address 127.7.17.254:35707)
0927 15:09:03.085533 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:03.085538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:03.086281 (+ 743us) server_negotiation.cc:407] Connection header received
0927 15:09:03.086521 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:03.086525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:03.086582 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:03.086642 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:03.087711 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.088559 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.089835 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.090184 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.090260 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:03.092181 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:03.092185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:03.092188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:03.092223 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:03.094002 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:03.094566 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:03.094570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:03.094572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:03.094671 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:03.095027 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:03.095030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:03.095032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:03.095271 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:03.095382 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:03.095784 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 15:09:03.095920 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:09:03.100243 18466 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:09:03.104877 18466 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:09:03.134230 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:03.121719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57218 (local address 127.7.17.195:33631)
0927 15:09:03.123825 (+ 2106us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:03.123837 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:03.125095 (+ 1258us) server_negotiation.cc:407] Connection header received
0927 15:09:03.125181 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:03.125186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:03.125399 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:03.126476 (+ 1077us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:09:03.126898 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.128073 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.131065 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.132042 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.132174 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:03.132371 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:03.133917 (+ 1546us) server_negotiation.cc:299] Negotiation successful
0927 15:09:03.134052 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1986,"thread_start_us":644,"threads_started":1}
I20260927 15:09:03.134733 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:03.121275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:33631 (local address 127.0.0.1:57218)
0927 15:09:03.124898 (+ 3623us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:03.124913 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:03.125001 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:03.126636 (+ 1635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:03.126639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:03.126656 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:09:03.126755 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:03.126762 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.128212 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.128238 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:03.130813 (+ 2575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:03.130853 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.132456 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.132460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:03.132604 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:03.132650 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 15:09:03.132771 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 15:09:03.134595 (+ 1824us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3510,"thread_start_us":88,"threads_started":1}
I20260927 15:09:03.137482 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:03.118127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:46861 (local address 127.0.0.1:53086)
0927 15:09:03.124122 (+ 5995us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:03.124192 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:03.124333 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:03.125355 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:03.125361 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:03.125385 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:09:03.125569 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:03.125577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.133913 (+ 8336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.133917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:03.135113 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:03.135126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.137077 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.137081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:03.137177 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:03.137201 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:09:03.137269 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 15:09:03.137345 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5795,"thread_start_us":116,"threads_started":1}
I20260927 15:09:03.138587 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:03.118248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53086 (local address 127.7.17.193:46861)
0927 15:09:03.124528 (+ 6280us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:03.124541 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:03.124570 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:09:03.124650 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:03.124656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:03.124970 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:03.125206 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:09:03.125766 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.133746 (+ 7980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.135810 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.136679 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.136807 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:03.137009 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:03.138347 (+ 1338us) server_negotiation.cc:299] Negotiation successful
0927 15:09:03.138457 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6162,"thread_start_us":101,"threads_started":1}
I20260927 15:09:03.138545 18887 tablet_service.cc:1467] Processing CreateTablet for tablet cdc69d8d86da4838adfaaac4e7901409 (DEFAULT_TABLE table=test-table [id=920cb7813c0c425fa478869b5cd3f599]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:09:03.139767 18887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc69d8d86da4838adfaaac4e7901409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:03.141788 18618 tablet_service.cc:1467] Processing CreateTablet for tablet cdc69d8d86da4838adfaaac4e7901409 (DEFAULT_TABLE table=test-table [id=920cb7813c0c425fa478869b5cd3f599]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:09:03.142912 18618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc69d8d86da4838adfaaac4e7901409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:03.157821 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:03.120514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:38425 (local address 127.0.0.1:46760)
0927 15:09:03.124576 (+ 4062us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:03.124589 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:03.124667 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:03.150221 (+ 25554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:03.150225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:03.150253 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:09:03.150386 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:03.150396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.153667 (+ 3271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.153672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:03.154931 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:03.154947 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.157333 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.157337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:03.157447 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:03.157478 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 15:09:03.157553 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 15:09:03.157650 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3998,"thread_start_us":111,"threads_started":1}
I20260927 15:09:03.158524 18973 tablet_bootstrap.cc:492] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe: Bootstrap starting.
I20260927 15:09:03.162169 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:03.120683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46760 (local address 127.7.17.194:38425)
0927 15:09:03.135423 (+ 14740us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:03.135436 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:03.135462 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:09:03.135541 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:03.135547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:03.138009 (+ 2462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:03.138172 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:09:03.150575 (+ 12403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.153498 (+ 2923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.155137 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.156027 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.156139 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:03.161802 (+ 5663us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:03.161882 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 15:09:03.162006 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14617,"thread_start_us":142,"threads_started":1}
I20260927 15:09:03.163647 18974 tablet_bootstrap.cc:492] T cdc69d8d86da4838adfaaac4e7901409 P d0dbb6edef6a4f4dbe8377edff7bde77: Bootstrap starting.
I20260927 15:09:03.165552 18736 tablet_service.cc:1467] Processing CreateTablet for tablet cdc69d8d86da4838adfaaac4e7901409 (DEFAULT_TABLE table=test-table [id=920cb7813c0c425fa478869b5cd3f599]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:09:03.167018 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc69d8d86da4838adfaaac4e7901409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:03.167770 18974 tablet_bootstrap.cc:654] T cdc69d8d86da4838adfaaac4e7901409 P d0dbb6edef6a4f4dbe8377edff7bde77: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:03.169377 18974 log.cc:824] T cdc69d8d86da4838adfaaac4e7901409 P d0dbb6edef6a4f4dbe8377edff7bde77: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:03.171911 18974 tablet_bootstrap.cc:492] T cdc69d8d86da4838adfaaac4e7901409 P d0dbb6edef6a4f4dbe8377edff7bde77: No bootstrap required, opened a new log
I20260927 15:09:03.172165 18974 ts_tablet_manager.cc:1397] T cdc69d8d86da4838adfaaac4e7901409 P d0dbb6edef6a4f4dbe8377edff7bde77: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 15:09:03.174665 18973 tablet_bootstrap.cc:654] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:03.175741 18973 log.cc:824] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:03.180644 18974 raft_consensus.cc:348] T cdc69d8d86da4838adfaaac4e7901409 P d0dbb6edef6a4f4dbe8377edff7bde77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63386367abbc4ea1bf457560cdbb96fe" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 46861 } } peers { permanent_uuid: "b44f42d7b14540ceb9791e13b173b14c" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 38425 } } peers { permanent_uuid: "d0dbb6edef6a4f4dbe8377edff7bde77" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 33631 } }
I20260927 15:09:03.181053 18974 raft_consensus.cc:374] T cdc69d8d86da4838adfaaac4e7901409 P d0dbb6edef6a4f4dbe8377edff7bde77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:03.181124 18974 raft_consensus.cc:729] T cdc69d8d86da4838adfaaac4e7901409 P d0dbb6edef6a4f4dbe8377edff7bde77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbb6edef6a4f4dbe8377edff7bde77, State: Initialized, Role: FOLLOWER
I20260927 15:09:03.181869 18974 consensus_queue.cc:260] T cdc69d8d86da4838adfaaac4e7901409 P d0dbb6edef6a4f4dbe8377edff7bde77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63386367abbc4ea1bf457560cdbb96fe" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 46861 } } peers { permanent_uuid: "b44f42d7b14540ceb9791e13b173b14c" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 38425 } } peers { permanent_uuid: "d0dbb6edef6a4f4dbe8377edff7bde77" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 33631 } }
I20260927 15:09:03.184590 18974 ts_tablet_manager.cc:1428] T cdc69d8d86da4838adfaaac4e7901409 P d0dbb6edef6a4f4dbe8377edff7bde77: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260927 15:09:03.185094 18954 heartbeater.cc:503] Master 127.7.17.254:35707 was elected leader, sending a full tablet report...
I20260927 15:09:03.186851 18973 tablet_bootstrap.cc:492] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe: No bootstrap required, opened a new log
I20260927 15:09:03.187142 18973 ts_tablet_manager.cc:1397] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.000s
I20260927 15:09:03.189769 18976 tablet_bootstrap.cc:492] T cdc69d8d86da4838adfaaac4e7901409 P b44f42d7b14540ceb9791e13b173b14c: Bootstrap starting.
I20260927 15:09:03.192530 18976 tablet_bootstrap.cc:654] T cdc69d8d86da4838adfaaac4e7901409 P b44f42d7b14540ceb9791e13b173b14c: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:03.192366 18973 raft_consensus.cc:348] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63386367abbc4ea1bf457560cdbb96fe" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 46861 } } peers { permanent_uuid: "b44f42d7b14540ceb9791e13b173b14c" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 38425 } } peers { permanent_uuid: "d0dbb6edef6a4f4dbe8377edff7bde77" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 33631 } }
I20260927 15:09:03.192785 18973 raft_consensus.cc:374] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:03.192903 18973 raft_consensus.cc:729] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63386367abbc4ea1bf457560cdbb96fe, State: Initialized, Role: FOLLOWER
I20260927 15:09:03.193475 18973 consensus_queue.cc:260] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63386367abbc4ea1bf457560cdbb96fe" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 46861 } } peers { permanent_uuid: "b44f42d7b14540ceb9791e13b173b14c" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 38425 } } peers { permanent_uuid: "d0dbb6edef6a4f4dbe8377edff7bde77" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 33631 } }
I20260927 15:09:03.194648 18973 ts_tablet_manager.cc:1428] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260927 15:09:03.195061 18684 heartbeater.cc:503] Master 127.7.17.254:35707 was elected leader, sending a full tablet report...
I20260927 15:09:03.200870 18976 log.cc:824] T cdc69d8d86da4838adfaaac4e7901409 P b44f42d7b14540ceb9791e13b173b14c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:03.203667 18976 tablet_bootstrap.cc:492] T cdc69d8d86da4838adfaaac4e7901409 P b44f42d7b14540ceb9791e13b173b14c: No bootstrap required, opened a new log
I20260927 15:09:03.203918 18976 ts_tablet_manager.cc:1397] T cdc69d8d86da4838adfaaac4e7901409 P b44f42d7b14540ceb9791e13b173b14c: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260927 15:09:03.207975 18976 raft_consensus.cc:348] T cdc69d8d86da4838adfaaac4e7901409 P b44f42d7b14540ceb9791e13b173b14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63386367abbc4ea1bf457560cdbb96fe" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 46861 } } peers { permanent_uuid: "b44f42d7b14540ceb9791e13b173b14c" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 38425 } } peers { permanent_uuid: "d0dbb6edef6a4f4dbe8377edff7bde77" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 33631 } }
I20260927 15:09:03.208305 18976 raft_consensus.cc:374] T cdc69d8d86da4838adfaaac4e7901409 P b44f42d7b14540ceb9791e13b173b14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:03.208350 18976 raft_consensus.cc:729] T cdc69d8d86da4838adfaaac4e7901409 P b44f42d7b14540ceb9791e13b173b14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44f42d7b14540ceb9791e13b173b14c, State: Initialized, Role: FOLLOWER
I20260927 15:09:03.209828 18976 consensus_queue.cc:260] T cdc69d8d86da4838adfaaac4e7901409 P b44f42d7b14540ceb9791e13b173b14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63386367abbc4ea1bf457560cdbb96fe" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 46861 } } peers { permanent_uuid: "b44f42d7b14540ceb9791e13b173b14c" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 38425 } } peers { permanent_uuid: "d0dbb6edef6a4f4dbe8377edff7bde77" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 33631 } }
I20260927 15:09:03.211051 18976 ts_tablet_manager.cc:1428] T cdc69d8d86da4838adfaaac4e7901409 P b44f42d7b14540ceb9791e13b173b14c: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 15:09:03.211484 18820 heartbeater.cc:503] Master 127.7.17.254:35707 was elected leader, sending a full tablet report...
I20260927 15:09:03.262679 18979 raft_consensus.cc:482] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
Sep 27 15:09:03 dist-test-slave-hhpq krb5kdc[17994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521742, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for HTTP/127.7.17.212@KRBTEST.COM
I20260927 15:09:03.262917 18979 raft_consensus.cc:504] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63386367abbc4ea1bf457560cdbb96fe" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 46861 } } peers { permanent_uuid: "b44f42d7b14540ceb9791e13b173b14c" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 38425 } } peers { permanent_uuid: "d0dbb6edef6a4f4dbe8377edff7bde77" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 33631 } }
I20260927 15:09:03.264209 18979 leader_election.cc:288] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b44f42d7b14540ceb9791e13b173b14c (127.7.17.194:38425), d0dbb6edef6a4f4dbe8377edff7bde77 (127.7.17.195:33631)
I20260927 15:09:03.276516 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:03.265092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:41561 (local address 127.7.17.195:33631)
0927 15:09:03.265218 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:03.265222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:03.266531 (+ 1309us) server_negotiation.cc:407] Connection header received
0927 15:09:03.270158 (+ 3627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:03.270162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:03.270226 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:03.270278 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:09:03.270677 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.271405 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.273924 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.274523 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.274635 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:03.274693 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:03.276272 (+ 1579us) server_negotiation.cc:299] Negotiation successful
0927 15:09:03.276371 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:09:03.277308 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:03.265303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:45547 (local address 127.7.17.194:38425)
0927 15:09:03.265441 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:03.265446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:03.265653 (+ 207us) server_negotiation.cc:407] Connection header received
0927 15:09:03.265870 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:03.265873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:03.265924 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:03.265971 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:09:03.266903 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.267640 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.275172 (+ 7532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.275758 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.275872 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:03.275926 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:03.277091 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0927 15:09:03.277185 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:09:03.277742 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:03.264579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:38425 (local address 127.7.17.193:45547)
0927 15:09:03.265552 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:03.265567 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:03.265745 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:03.266102 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:03.266105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:03.266122 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:09:03.266651 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:03.266659 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.267775 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.267778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:03.275033 (+ 7255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:03.275040 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.276024 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.276027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:03.276131 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:03.276152 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:09:03.277500 (+ 1348us) client_negotiation.cc:240] Negotiation successful
0927 15:09:03.277582 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"thread_start_us":108,"threads_started":1}
I20260927 15:09:03.279984 18766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc69d8d86da4838adfaaac4e7901409" candidate_uuid: "63386367abbc4ea1bf457560cdbb96fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44f42d7b14540ceb9791e13b173b14c" is_pre_election: true
I20260927 15:09:03.280366 18766 raft_consensus.cc:2454] T cdc69d8d86da4838adfaaac4e7901409 P b44f42d7b14540ceb9791e13b173b14c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63386367abbc4ea1bf457560cdbb96fe in term 0.
I20260927 15:09:03.281000 18570 leader_election.cc:302] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63386367abbc4ea1bf457560cdbb96fe, b44f42d7b14540ceb9791e13b173b14c; no voters:
I20260927 15:09:03.281347 18979 raft_consensus.cc:2781] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:09:03.281440 18979 raft_consensus.cc:482] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:09:03.281503 18979 raft_consensus.cc:3037] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:03.282864 18979 raft_consensus.cc:504] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63386367abbc4ea1bf457560cdbb96fe" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 46861 } } peers { permanent_uuid: "b44f42d7b14540ceb9791e13b173b14c" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 38425 } } peers { permanent_uuid: "d0dbb6edef6a4f4dbe8377edff7bde77" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 33631 } }
I20260927 15:09:03.283241 18979 leader_election.cc:288] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe [CANDIDATE]: Term 1 election: Requested vote from peers b44f42d7b14540ceb9791e13b173b14c (127.7.17.194:38425), d0dbb6edef6a4f4dbe8377edff7bde77 (127.7.17.195:33631)
I20260927 15:09:03.284745 18766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc69d8d86da4838adfaaac4e7901409" candidate_uuid: "63386367abbc4ea1bf457560cdbb96fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44f42d7b14540ceb9791e13b173b14c"
I20260927 15:09:03.284924 18766 raft_consensus.cc:3037] T cdc69d8d86da4838adfaaac4e7901409 P b44f42d7b14540ceb9791e13b173b14c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:03.286279 18766 raft_consensus.cc:2454] T cdc69d8d86da4838adfaaac4e7901409 P b44f42d7b14540ceb9791e13b173b14c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63386367abbc4ea1bf457560cdbb96fe in term 1.
I20260927 15:09:03.286695 18570 leader_election.cc:302] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63386367abbc4ea1bf457560cdbb96fe, b44f42d7b14540ceb9791e13b173b14c; no voters:
I20260927 15:09:03.286916 18979 raft_consensus.cc:2781] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:09:03.287139 18979 raft_consensus.cc:686] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe [term 1 LEADER]: Becoming Leader. State: Replica: 63386367abbc4ea1bf457560cdbb96fe, State: Running, Role: LEADER
I20260927 15:09:03.287550 18979 consensus_queue.cc:237] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63386367abbc4ea1bf457560cdbb96fe" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 46861 } } peers { permanent_uuid: "b44f42d7b14540ceb9791e13b173b14c" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 38425 } } peers { permanent_uuid: "d0dbb6edef6a4f4dbe8377edff7bde77" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 33631 } }
I20260927 15:09:03.291347 18466 catalog_manager.cc:5495] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 63386367abbc4ea1bf457560cdbb96fe (127.7.17.193). New cstate: current_term: 1 leader_uuid: "63386367abbc4ea1bf457560cdbb96fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63386367abbc4ea1bf457560cdbb96fe" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 46861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b44f42d7b14540ceb9791e13b173b14c" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0dbb6edef6a4f4dbe8377edff7bde77" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 33631 } health_report { overall_health: UNKNOWN } } }
I20260927 15:09:03.292714 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:03.264958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:33631 (local address 127.7.17.193:41561)
0927 15:09:03.266380 (+ 1422us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:03.266393 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:03.266792 (+ 399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:03.270409 (+ 3617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:03.270412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:03.270432 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:09:03.270551 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:03.270557 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.271563 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.271566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:03.273329 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:03.273342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.274807 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.274810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:03.274918 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:03.274943 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:09:03.276760 (+ 1817us) client_negotiation.cc:240] Negotiation successful
0927 15:09:03.276890 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1361,"thread_start_us":78,"threads_started":1}
I20260927 15:09:03.293514 18892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc69d8d86da4838adfaaac4e7901409" candidate_uuid: "63386367abbc4ea1bf457560cdbb96fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbb6edef6a4f4dbe8377edff7bde77" is_pre_election: true
I20260927 15:09:03.293854 18892 raft_consensus.cc:2454] T cdc69d8d86da4838adfaaac4e7901409 P d0dbb6edef6a4f4dbe8377edff7bde77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63386367abbc4ea1bf457560cdbb96fe in term 0.
I20260927 15:09:03.294229 18905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc69d8d86da4838adfaaac4e7901409" candidate_uuid: "63386367abbc4ea1bf457560cdbb96fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbb6edef6a4f4dbe8377edff7bde77"
I20260927 15:09:03.294350 18905 raft_consensus.cc:3037] T cdc69d8d86da4838adfaaac4e7901409 P d0dbb6edef6a4f4dbe8377edff7bde77 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:03.295755 18905 raft_consensus.cc:2454] T cdc69d8d86da4838adfaaac4e7901409 P d0dbb6edef6a4f4dbe8377edff7bde77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63386367abbc4ea1bf457560cdbb96fe in term 1.
W20260927 15:09:03.296064 18955 tablet.cc:2369] T cdc69d8d86da4838adfaaac4e7901409 P d0dbb6edef6a4f4dbe8377edff7bde77: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:09:03.315812 18685 tablet.cc:2369] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:09:03.342900 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:03.337426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53102 (local address 127.7.17.193:46861)
0927 15:09:03.337547 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:03.337551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:03.337563 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:09:03.337655 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:03.337658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:03.337723 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:03.337824 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:09:03.338215 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.339007 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.340520 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:03.340867 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:03.340960 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:03.341541 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:09:03.342115 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:09:03.342226 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:03.342673 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 15:09:03.342741 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:09:03.351181 18905 raft_consensus.cc:1261] T cdc69d8d86da4838adfaaac4e7901409 P d0dbb6edef6a4f4dbe8377edff7bde77 [term 1 FOLLOWER]: Refusing update from remote peer 63386367abbc4ea1bf457560cdbb96fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:09:03.352080 18985 consensus_queue.cc:1035] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbb6edef6a4f4dbe8377edff7bde77" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 33631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:09:03.360337 18766 raft_consensus.cc:1261] T cdc69d8d86da4838adfaaac4e7901409 P b44f42d7b14540ceb9791e13b173b14c [term 1 FOLLOWER]: Refusing update from remote peer 63386367abbc4ea1bf457560cdbb96fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:09:03.361308 18985 consensus_queue.cc:1035] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b44f42d7b14540ceb9791e13b173b14c" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:09:03.380789 18989 mvcc.cc:205] Tried to move back new op lower bound from 7333977060756496384 to 7333977060509609984. Current Snapshot: MvccSnapshot[applied={T|T < 7333977060756496384}]
I20260927 15:09:03.382460 18992 mvcc.cc:205] Tried to move back new op lower bound from 7333977060756496384 to 7333977060509609984. Current Snapshot: MvccSnapshot[applied={T|T < 7333977060756496384}]
I20260927 15:09:03.382642 18990 mvcc.cc:205] Tried to move back new op lower bound from 7333977060756496384 to 7333977060509609984. Current Snapshot: MvccSnapshot[applied={T|T < 7333977060756496384}]
I20260927 15:09:03.393615 18466 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:09:03.393797 18466 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:09:03.396519 18466 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1c24e56a4484031a3e653ebf2aa6ad1: Sending DeleteTablet for 3 replicas of tablet cdc69d8d86da4838adfaaac4e7901409
I20260927 15:09:03.397759 18887 tablet_service.cc:1514] Processing DeleteTablet for tablet cdc69d8d86da4838adfaaac4e7901409 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:09:03 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:57218
I20260927 15:09:03.398445 18618 tablet_service.cc:1514] Processing DeleteTablet for tablet cdc69d8d86da4838adfaaac4e7901409 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:09:03 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:53086
I20260927 15:09:03.398990 18736 tablet_service.cc:1514] Processing DeleteTablet for tablet cdc69d8d86da4838adfaaac4e7901409 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:09:03 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:46760
I20260927 15:09:03.400048 18999 tablet_replica.cc:330] T cdc69d8d86da4838adfaaac4e7901409 P 63386367abbc4ea1bf457560cdbb96fe: stopping tablet replica
I20260927 15:09:03.400305 19000 tablet_replica.cc:330] T cdc69d8d86da4838adfaaac4e7901409 P b44f42d7b14540ceb9791e13b173b14c: stopping tablet replica
I20260927 15:09:03.400558 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 18551
I20260927 15:09:03.400727 19000 raft_consensus.cc:2229] T cdc69d8d86da4838adfaaac4e7901409 P b44f42d7b14540ceb9791e13b173b14c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:09:03.401589 19000 raft_consensus.cc:2258] T cdc69d8d86da4838adfaaac4e7901409 P b44f42d7b14540ceb9791e13b173b14c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:09:03.403247 19000 ts_tablet_manager.cc:1905] T cdc69d8d86da4838adfaaac4e7901409 P b44f42d7b14540ceb9791e13b173b14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:09:03.406157 18998 tablet_replica.cc:330] T cdc69d8d86da4838adfaaac4e7901409 P d0dbb6edef6a4f4dbe8377edff7bde77: stopping tablet replica
I20260927 15:09:03.406203 19000 ts_tablet_manager.cc:1918] T cdc69d8d86da4838adfaaac4e7901409 P b44f42d7b14540ceb9791e13b173b14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:09:03.406292 19000 log.cc:1196] T cdc69d8d86da4838adfaaac4e7901409 P b44f42d7b14540ceb9791e13b173b14c: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-1/wal/wals/cdc69d8d86da4838adfaaac4e7901409
I20260927 15:09:03.406595 19000 ts_tablet_manager.cc:1939] T cdc69d8d86da4838adfaaac4e7901409 P b44f42d7b14540ceb9791e13b173b14c: Deleting consensus metadata
I20260927 15:09:03.407305 18998 raft_consensus.cc:2229] T cdc69d8d86da4838adfaaac4e7901409 P d0dbb6edef6a4f4dbe8377edff7bde77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:09:03.407717 18998 raft_consensus.cc:2258] T cdc69d8d86da4838adfaaac4e7901409 P d0dbb6edef6a4f4dbe8377edff7bde77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:09:03.408005 18455 catalog_manager.cc:4841] TS b44f42d7b14540ceb9791e13b173b14c (127.7.17.194:38425): tablet cdc69d8d86da4838adfaaac4e7901409 (table test-table [id=920cb7813c0c425fa478869b5cd3f599]) successfully deleted
I20260927 15:09:03.417258 18998 ts_tablet_manager.cc:1905] T cdc69d8d86da4838adfaaac4e7901409 P d0dbb6edef6a4f4dbe8377edff7bde77: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 15:09:03.420327 18456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.17.193:46861 (error 108)
I20260927 15:09:03.420643 18998 ts_tablet_manager.cc:1918] T cdc69d8d86da4838adfaaac4e7901409 P d0dbb6edef6a4f4dbe8377edff7bde77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:09:03.420758 18998 log.cc:1196] T cdc69d8d86da4838adfaaac4e7901409 P d0dbb6edef6a4f4dbe8377edff7bde77: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/cdc69d8d86da4838adfaaac4e7901409
I20260927 15:09:03.421484 18998 ts_tablet_manager.cc:1939] T cdc69d8d86da4838adfaaac4e7901409 P d0dbb6edef6a4f4dbe8377edff7bde77: Deleting consensus metadata
I20260927 15:09:03.421792 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 18690
I20260927 15:09:03.422578 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:03.421957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:46861 (local address 127.0.0.1:53114)
0927 15:09:03.422410 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:03.422485 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.193:46861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356}
W20260927 15:09:03.423377 18456 catalog_manager.cc:4584] TS 63386367abbc4ea1bf457560cdbb96fe (127.7.17.193:46861): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cdc69d8d86da4838adfaaac4e7901409: Network error: Client connection negotiation failed: client connection to 127.7.17.193:46861: connect: Connection refused (error 111)
I20260927 15:09:03.429695 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 18825
W20260927 15:09:03.438772 18969 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:09:03.438405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:33631 (local address 127.0.0.1:57224)
0927 15:09:03.438564 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:03.438637 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.195:33631: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 15:09:03.438946 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 18438
I20260927 15:09:03.450699 7239 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 15:09:03.785960 7239 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 15:09:03.786092 7239 mini_ranger.cc:66] Stopping Ranger...
I20260927 15:09:04.121584 7239 mini_ranger.cc:68] Stopped Ranger
2026-09-27 15:09:04.121 UTC [18024] LOG: received smart shutdown request
2026-09-27 15:09:04.125 UTC [18024] LOG: background worker "logical replication launcher" (PID 18033) exited with exit code 1
2026-09-27 15:09:04.125 UTC [18028] LOG: shutting down
2026-09-27 15:09:04.299 UTC [18024] LOG: database system is shut down
2026-09-27T15:09:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47197 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:09:04 dist-test-slave-hhpq krb5kdc[19010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:09:04 dist-test-slave-hhpq krb5kdc[19010](info): set up 2 sockets
Sep 27 15:09:04 dist-test-slave-hhpq krb5kdc[19010](info): commencing operation
krb5kdc: starting...
W20260927 15:09:05.353020 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:09:05 dist-test-slave-hhpq krb5kdc[19010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:09:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:09:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.17.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.17.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.17.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/rangeradmin_127.7.17.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.17.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/rangeradmin_127.7.17.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.17.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.17.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.17.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/rangerlookup_127.7.17.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.17.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/rangerlookup_127.7.17.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.17.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.17.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.17.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/HTTP_127.7.17.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.17.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/HTTP_127.7.17.212@KRBTEST.COM.keytab.
I20260927 15:09:05.519939 7239 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskgkjf1n/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/postgres -l logfile start
2026-09-27 15:09:08.048 UTC [19040] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 15:09:08.048 UTC [19040] LOG: listening on IPv4 address "127.7.17.212", port 60871
2026-09-27 15:09:08.051 UTC [19040] LOG: listening on Unix socket "/tmp/.s.PGSQL.60871"
2026-09-27 15:09:08.078 UTC [19043] LOG: database system was shut down at 2026-09-27 15:09:06 UTC
2026-09-27 15:09:08.082 UTC [19040] LOG: database system is ready to accept connections
I20260927 15:09:08.960548 7239 mini_postgres.cc:96] Postgres bound to 60871
2026-09-27 15:09:08.966 UTC [19051] FATAL: database "slave" does not exist
127.7.17.212:60871 - accepting connections
I20260927 15:09:08.967079 7239 mini_ranger.cc:161] Starting Ranger...
I20260927 15:09:08.981468 7239 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 15:09:09.477998 7239 mini_ranger.cc:87] Created ranger Postgres database
I20260927 15:09:09.478068 7239 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ranger-admin
2026-09-27 15:09:09,757 [I] DB FLAVOR :POSTGRES
2026-09-27 15:09:09,757 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:09:09,757 [I] Checking connection..
2026-09-27 15:09:09,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 15:09:10,017 [I] Checking connection passed.
2026-09-27 15:09:10,018 [I] --------- Verifying version history table ---------
2026-09-27 15:09:10,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:09:10,260 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:09:10,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:09:10,516 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:09:10,517 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 15:09:10,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 15:09:10.752 UTC [19115] WARNING: there is no transaction in progress
2026-09-27 15:09:10,759 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 15:09:10,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:09:11,008 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 15:09:11,008 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:09:11,009 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 15:09:11,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 15:09:11,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hhpq.c.gcp-upstream.internal','N') ;"
2026-09-27 15:09:11,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:09:11,971 [I] Table x_portal_user does not exist in database ranger
2026-09-27 15:09:11,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:09:12,232 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 15:09:12,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:09:12,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:09:12,721 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 15:09:12,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 15:09:13.760 UTC [19223] WARNING: there is no transaction in progress
2026-09-27 15:09:13.778 UTC [19223] WARNING: there is no transaction in progress
2026-09-27 15:09:13.797 UTC [19223] WARNING: there is no transaction in progress
2026-09-27 15:09:13.813 UTC [19223] WARNING: there is no transaction in progress
2026-09-27 15:09:13.832 UTC [19223] WARNING: there is no transaction in progress
2026-09-27 15:09:13.852 UTC [19223] WARNING: there is no transaction in progress
2026-09-27 15:09:13.910 UTC [19223] WARNING: there is no transaction in progress
2026-09-27 15:09:13.918 UTC [19223] WARNING: there is no transaction in progress
2026-09-27 15:09:13.935 UTC [19223] WARNING: there is no transaction in progress
2026-09-27 15:09:13.947 UTC [19223] WARNING: there is no transaction in progress
2026-09-27 15:09:13.958 UTC [19223] WARNING: there is no transaction in progress
2026-09-27 15:09:13.974 UTC [19223] WARNING: there is no transaction in progress
2026-09-27 15:09:13.985 UTC [19223] WARNING: there is no transaction in progress
2026-09-27 15:09:13.992 UTC [19223] WARNING: there is no transaction in progress
2026-09-27 15:09:14.000 UTC [19223] WARNING: there is no transaction in progress
2026-09-27 15:09:14.008 UTC [19223] WARNING: there is no transaction in progress
2026-09-27 15:09:14,738 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 15:09:14,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 15:09:14,991 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 15:09:14,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:09:15,224 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 15:09:15,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:09:15,483 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 15:09:15,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:09:15,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:09:15,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hhpq.c.gcp-upstream.internal';"
2026-09-27 15:09:16,231 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 15:09:16,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 15:09:16,487 [I] DB_PATCHES have already been applied
I20260927 15:09:16.492894 7239 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ranger-admin:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskgkjf1n/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 15:09:16.492965 7239 mini_ranger.cc:193] Using host: 127.7.17.212
I20260927 15:09:16.496356 7239 mini_ranger.cc:227] Ranger admin URL: http://127.7.17.212:45967
Sep 27, 2026 3:09:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:09:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:09:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 3:09:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 3:09:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 3:09:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 3:09:17 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 15:09:17.506834 7239 mini_ranger.cc:160] Time spent starting Ranger: real 8.540s user 0.003s sys 0.007s
Sep 27, 2026 3:09:38 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 3:09:38 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 3:09:38 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 3:09:38 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 3:09:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 3:09:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 3:09:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 3:09:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 3:09:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 3:09:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 3:09:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 3:09:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 3:09:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 3:09:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 3:09:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 3:09:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 3:09:38 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 3:09:39 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 15:09:40.082180 7239 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.17.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.17.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.17.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/rangerkms_127.7.17.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.17.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/rangerkms_127.7.17.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.17.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.17.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.17.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/HTTP_127.7.17.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.17.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/HTTP_127.7.17.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 15:09:40 dist-test-slave-hhpq krb5kdc[19010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521780, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 15:09:40.167279 7239 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 15:09:40.177203 7239 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 15:09:41.104270 7239 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 15:09:41.104351 7239 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ranger-kms
I20260927 15:09:41.104373 7239 mini_ranger_kms.cc:227] Using postgres at 127.7.17.212:60871
2026-09-27 15:09:41,261 [I] DB FLAVOR :POSTGRES
2026-09-27 15:09:41,262 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:09:41,262 [I] Checking connection
2026-09-27 15:09:41,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:60871/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 15:09:41,527 [I] connection success
2026-09-27 15:09:41,527 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 15:09:41,527 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 15:09:41,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:60871/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 15:09:41,770 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 15:09:41,771 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:09:41,771 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 15:09:41,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.17.212:60871/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 15:09:42,027 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 15:09:42.568351 7239 mini_ranger_kms.cc:310] Created kms service
I20260927 15:09:42.753989 7239 mini_ranger_kms.cc:326] Created kudu user
I20260927 15:09:42.797912 7239 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 15:09:42 dist-test-slave-hhpq krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521780, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.17.212@KRBTEST.COM
I20260927 15:09:43.196475 7239 mini_ranger_kms.cc:384] Added ranger policy
I20260927 15:09:43.196722 7239 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ranger-kms:/tmp/dist-test-taskgkjf1n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskgkjf1n/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 15:09:43.196746 7239 mini_ranger_kms.cc:240] Using host: 127.7.17.212
I20260927 15:09:43.200698 7239 mini_ranger_kms.cc:276] Ranger KMS PID: 19403
I20260927 15:09:43.200783 7239 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.17.212:43531
Sep 27, 2026 3:09:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:09:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:09:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 3:09:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 3:09:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 3:09:44 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 15:09:45.256414 7239 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.089s user 0.014s sys 0.127s
I20260927 15:09:45.256559 7239 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 15:09:45.256628 7239 mini_ranger_kms.cc:401] 127.7.17.212:43531/kms/v1/keys
Sep 27 15:09:45 dist-test-slave-hhpq krb5kdc[19010](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790521785, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.17.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 15:09:45 dist-test-slave-hhpq krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521785, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.17.212@KRBTEST.COM for HTTP/127.7.17.212@KRBTEST.COM
Sep 27 15:09:46 dist-test-slave-hhpq krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521785, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.17.212@KRBTEST.COM for HTTP/127.7.17.212@KRBTEST.COM
Sep 27 15:09:48 dist-test-slave-hhpq krb5kdc[19010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:09:48.347724 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:43727
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:43727
--ranger_config_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.17.212:43531/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:09:48.465984 19454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:48.466269 19454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:48.466334 19454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:48.470501 19454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:09:48.470588 19454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:48.470614 19454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:48.470634 19454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:09:48.470655 19454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:09:48.476634 19454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.17.212:43531/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:43727
--ranger_config_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:43727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:09:48.477805 19454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:48.478896 19454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:48.487654 19460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:48.487655 19462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:48.487655 19459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:48.487959 19454 server_base.cc:1034] running on GCE node
I20260927 15:09:48.488664 19454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:48.489904 19454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:48.491070 19454 hybrid_clock.cc:648] HybridClock initialized: now 1790521788491055 us; error 32 us; skew 500 ppm
Sep 27 15:09:48 dist-test-slave-hhpq krb5kdc[19010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521788, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:48.493922 19454 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:09:48.494827 19454 webserver.cc:460] Webserver started at http://127.7.17.254:36881/ using document root <none> and password file <none>
I20260927 15:09:48.495359 19454 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:48.495436 19454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:48.495630 19454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:09:48 dist-test-slave-hhpq krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521788, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for HTTP/127.7.17.212@KRBTEST.COM
I20260927 15:09:48.546675 19454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "d8d3311eb7c04023b7256bb40c557a45"
format_stamp: "Formatted at 2026-09-27 15:09:48 on dist-test-slave-hhpq"
server_key: "428700d88bed91b47961df8adc242ef2"
server_key_iv: "4022b313d5513772b390d212704084a8"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:48.547343 19454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "d8d3311eb7c04023b7256bb40c557a45"
format_stamp: "Formatted at 2026-09-27 15:09:48 on dist-test-slave-hhpq"
server_key: "428700d88bed91b47961df8adc242ef2"
server_key_iv: "4022b313d5513772b390d212704084a8"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:48.551424 19454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 15:09:48.570125 19469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:48.571962 19454 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:09:48.572127 19454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "d8d3311eb7c04023b7256bb40c557a45"
format_stamp: "Formatted at 2026-09-27 15:09:48 on dist-test-slave-hhpq"
server_key: "428700d88bed91b47961df8adc242ef2"
server_key_iv: "4022b313d5513772b390d212704084a8"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:48.572279 19454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:48.613413 19454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:48.616394 19454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:48.616557 19454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:48.623474 19454 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:43727
I20260927 15:09:48.623492 19521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:43727 every 8 connection(s)
I20260927 15:09:48.624470 19454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:09:48.626138 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 19454
I20260927 15:09:48.626257 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:09:48.626565 7239 external_mini_cluster.cc:1428] Setting key 68ad2af2a1c7bb9e534bf5a0f60e04d8
I20260927 15:09:48.627653 19522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:48.633417 19522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45: Bootstrap starting.
Sep 27 15:09:48 dist-test-slave-hhpq krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:09:48.635879 19522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:48.636613 19522 log.cc:824] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:48.638686 19522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45: No bootstrap required, opened a new log
I20260927 15:09:48.641655 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:48.628103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57774 (local address 127.7.17.254:43727)
0927 15:09:48.628536 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:48.628552 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:48.628588 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:09:48.629317 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:48.629335 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:48.629684 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:48.630072 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:48.630840 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:48.631846 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:48.632708 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:48.633246 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:48.633390 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:48.636071 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:48.636077 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:48.636099 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:48.636131 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:48.638416 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:48.638894 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:48.638900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:48.638907 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:48.639056 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:48.639322 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:48.639326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:48.639327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:48.639867 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:48.640062 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:48.640341 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 15:09:48.640635 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":146,"threads_started":1}
I20260927 15:09:48.642652 19522 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d3311eb7c04023b7256bb40c557a45" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43727 } }
I20260927 15:09:48.642930 19522 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:48.642980 19522 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d3311eb7c04023b7256bb40c557a45, State: Initialized, Role: FOLLOWER
I20260927 15:09:48.643435 19522 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d3311eb7c04023b7256bb40c557a45" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43727 } }
I20260927 15:09:48.643556 19522 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:09:48.643612 19522 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:09:48.643704 19522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:48.644675 19522 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d3311eb7c04023b7256bb40c557a45" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43727 } }
I20260927 15:09:48.645031 19522 leader_election.cc:302] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8d3311eb7c04023b7256bb40c557a45; no voters:
I20260927 15:09:48.645299 19522 leader_election.cc:288] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:09:48.645442 19527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:09:48.645663 19527 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [term 1 LEADER]: Becoming Leader. State: Replica: d8d3311eb7c04023b7256bb40c557a45, State: Running, Role: LEADER
I20260927 15:09:48.646031 19527 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d3311eb7c04023b7256bb40c557a45" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43727 } }
I20260927 15:09:48.646544 19522 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:09:48.647624 19529 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8d3311eb7c04023b7256bb40c557a45. Latest consensus state: current_term: 1 leader_uuid: "d8d3311eb7c04023b7256bb40c557a45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d3311eb7c04023b7256bb40c557a45" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43727 } } }
I20260927 15:09:48.647748 19529 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [sys.catalog]: This master's current role is: LEADER
I20260927 15:09:48.648096 19528 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8d3311eb7c04023b7256bb40c557a45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d3311eb7c04023b7256bb40c557a45" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43727 } } }
I20260927 15:09:48.648177 19536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:09:48.648216 19528 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [sys.catalog]: This master's current role is: LEADER
I20260927 15:09:48.648684 19522 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 15:09:48.653883 19536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:09:48.662441 19536 catalog_manager.cc:1347] Generated new cluster ID: 14608b39d73244e68691b4f82dea8d4b
I20260927 15:09:48.662523 19536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:09:48.697664 19536 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:09:48.698683 19536 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:09:48.702292 19536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45: Generated new TSK 0
I20260927 15:09:48.703004 19536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
I20260927 15:09:49.408274 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.17.212:43531/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43727
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:09:49.545535 19567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:49.545840 19567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:49.545883 19567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:49.550318 19567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:49.550401 19567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:49.550468 19567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:09:49.557605 19567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.17.212:43531/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:09:49.559609 19567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:49.561228 19567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:49.573681 19574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:49.573906 19567 server_base.cc:1034] running on GCE node
W20260927 15:09:49.574030 19575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:49.574189 19577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:49.574697 19567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:49.575534 19567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:49.576774 19567 hybrid_clock.cc:648] HybridClock initialized: now 1790521789576738 us; error 54 us; skew 500 ppm
Sep 27 15:09:49 dist-test-slave-hhpq krb5kdc[19010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521789, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:49.580363 19567 init.cc:377] Logged in from keytab as kudu/127.7.17.193@KRBTEST.COM (short username kudu)
I20260927 15:09:49.581514 19567 webserver.cc:460] Webserver started at http://127.7.17.193:34697/ using document root <none> and password file <none>
I20260927 15:09:49.582111 19567 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:49.582186 19567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:49.582424 19567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:09:49 dist-test-slave-hhpq krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521789, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for HTTP/127.7.17.212@KRBTEST.COM
I20260927 15:09:49.603266 19567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "4adb5b1cee8640b5bfc3ae6024c2921c"
format_stamp: "Formatted at 2026-09-27 15:09:49 on dist-test-slave-hhpq"
server_key: "bce4a654995b3c3f8e95573befeb10fa"
server_key_iv: "d80e4477acdb563584bdc3eae088e2c0"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:49.604095 19567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "4adb5b1cee8640b5bfc3ae6024c2921c"
format_stamp: "Formatted at 2026-09-27 15:09:49 on dist-test-slave-hhpq"
server_key: "bce4a654995b3c3f8e95573befeb10fa"
server_key_iv: "d80e4477acdb563584bdc3eae088e2c0"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:49.609560 19567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:09:49.644897 19584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:49.646828 19567 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.000s
I20260927 15:09:49.647001 19567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "4adb5b1cee8640b5bfc3ae6024c2921c"
format_stamp: "Formatted at 2026-09-27 15:09:49 on dist-test-slave-hhpq"
server_key: "bce4a654995b3c3f8e95573befeb10fa"
server_key_iv: "d80e4477acdb563584bdc3eae088e2c0"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:49.647163 19567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:49.680521 19567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:49.684525 19567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:49.684810 19567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:49.685883 19567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:09:49.686992 19567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:09:49.687105 19567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:49.687232 19567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:09:49.687309 19567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:49.699132 19567 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:38227
I20260927 15:09:49.700706 19567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:09:49.701447 19699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:38227 every 8 connection(s)
I20260927 15:09:49.706357 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 19567
I20260927 15:09:49.706472 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:09:49.706760 7239 external_mini_cluster.cc:1428] Setting key 96ce8c7eb3711615a4bf7d11c5c13ad0
Sep 27 15:09:49 dist-test-slave-hhpq krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521789, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.17.194@KRBTEST.COM; defaulting to no policy
I20260927 15:09:49.721856 19703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:49.704819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:43727 (local address 127.7.17.193:48763)
0927 15:09:49.705469 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:49.705539 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:49.706766 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:49.708225 (+ 1459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:49.708236 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:49.708608 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:49.709334 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:49.709355 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:49.710544 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:49.710547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:49.712041 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:49.712054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:49.712940 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:49.712944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:49.713178 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:49.713931 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:49.713954 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:49.715751 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:49.718159 (+ 2408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:49.718167 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:49.718197 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:49.718598 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:49.718988 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:49.718991 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:49.718993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:49.719214 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:49.719715 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:49.719722 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:49.720099 (+ 377us) client_negotiation.cc:769] Sending connection context
0927 15:09:49.720303 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 15:09:49.720672 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":147,"threads_started":1}
I20260927 15:09:49.722849 19702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:49.704305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:48763 (local address 127.7.17.254:43727)
0927 15:09:49.704604 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:49.704609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:49.705665 (+ 1056us) server_negotiation.cc:407] Connection header received
0927 15:09:49.707059 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:49.707064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:49.707125 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:49.707189 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:49.709530 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:49.710356 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:49.712209 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:49.712582 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:49.712698 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:49.716037 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:49.716041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:49.716044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:49.716077 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:49.717981 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:49.718738 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:49.718743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:49.718745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:49.718867 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:49.719324 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:49.719327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:49.719329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:49.719570 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:49.719660 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:49.722494 (+ 2834us) server_negotiation.cc:299] Negotiation successful
0927 15:09:49.722648 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.194@KRBTEST.COM" created.
I20260927 15:09:49.724239 19700 heartbeater.cc:347] Connected to a master server at 127.7.17.254:43727
I20260927 15:09:49.724534 19700 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:49.725274 19700 heartbeater.cc:511] Master 127.7.17.254:43727 requested a full tablet report, sending...
I20260927 15:09:49.727265 19486 ts_manager.cc:194] Registered new tserver with Master: 4adb5b1cee8640b5bfc3ae6024c2921c (127.7.17.193:38227)
I20260927 15:09:49.728958 19486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:48763
Sep 27 15:09:49 dist-test-slave-hhpq krb5kdc[19010](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790521789, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
I20260927 15:09:49.744684 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.17.212:43531/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43727
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:09:49.909965 19706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:49.910374 19706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:49.910439 19706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:49.916862 19706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:49.916963 19706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:49.917057 19706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:09:49.925681 19706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.17.212:43531/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:09:49.927302 19706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:49.928754 19706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:49.941131 19713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:49.941622 19715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:49.941740 19706 server_base.cc:1034] running on GCE node
W20260927 15:09:49.941783 19712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:49.942247 19706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:49.943078 19706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:49.944278 19706 hybrid_clock.cc:648] HybridClock initialized: now 1790521789944232 us; error 63 us; skew 500 ppm
Sep 27 15:09:49 dist-test-slave-hhpq krb5kdc[19010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521789, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:49.947911 19706 init.cc:377] Logged in from keytab as kudu/127.7.17.194@KRBTEST.COM (short username kudu)
I20260927 15:09:49.949095 19706 webserver.cc:460] Webserver started at http://127.7.17.194:35741/ using document root <none> and password file <none>
I20260927 15:09:49.949882 19706 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:49.950042 19706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:49.950361 19706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:09:49 dist-test-slave-hhpq krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521789, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for HTTP/127.7.17.212@KRBTEST.COM
I20260927 15:09:49.966187 19706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "80a00b28a3794cc99357cd941979bda5"
format_stamp: "Formatted at 2026-09-27 15:09:49 on dist-test-slave-hhpq"
server_key: "34a3a30e25b75a79ef98651af0e8bf48"
server_key_iv: "97dac25247fd08fa521501a815b4ddb5"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:49.966799 19706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "80a00b28a3794cc99357cd941979bda5"
format_stamp: "Formatted at 2026-09-27 15:09:49 on dist-test-slave-hhpq"
server_key: "34a3a30e25b75a79ef98651af0e8bf48"
server_key_iv: "97dac25247fd08fa521501a815b4ddb5"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:49.971596 19706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:09:49.988518 19722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:49.990031 19706 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:09:49.990156 19706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "80a00b28a3794cc99357cd941979bda5"
format_stamp: "Formatted at 2026-09-27 15:09:49 on dist-test-slave-hhpq"
server_key: "34a3a30e25b75a79ef98651af0e8bf48"
server_key_iv: "97dac25247fd08fa521501a815b4ddb5"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:49.990298 19706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:50.009919 19706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:50.013895 19706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:50.014171 19706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:50.015003 19706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:09:50.016127 19706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:09:50.016179 19706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:50.016244 19706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:09:50.016276 19706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:50.027758 19706 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:34647
I20260927 15:09:50.027797 19835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:34647 every 8 connection(s)
I20260927 15:09:50.028899 19706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:09:50.032684 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 19706
I20260927 15:09:50.032804 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:09:50.033144 7239 external_mini_cluster.cc:1428] Setting key 1e8989240f9d7053c5b24f30dac29562
WARNING: no policy specified for kudu/127.7.17.195@KRBTEST.COM; defaulting to no policy
Sep 27 15:09:50 dist-test-slave-hhpq krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521789, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.195@KRBTEST.COM" created.
I20260927 15:09:50.058298 19702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:50.033622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:39571 (local address 127.7.17.254:43727)
0927 15:09:50.033771 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:50.033776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:50.040429 (+ 6653us) server_negotiation.cc:407] Connection header received
0927 15:09:50.041957 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:50.041961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:50.042027 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:50.042089 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:50.044015 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.044889 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.046888 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.047251 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.047358 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:50.053165 (+ 5807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:50.053171 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:50.053174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:50.053213 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:50.055275 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:50.056081 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:50.056085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:50.056088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:50.056186 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:50.056671 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:50.056674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:50.056676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:50.056955 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:50.057032 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:50.057931 (+ 899us) server_negotiation.cc:299] Negotiation successful
0927 15:09:50.058117 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:09:50.060389 19838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:50.032296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:43727 (local address 127.7.17.194:39571)
0927 15:09:50.040184 (+ 7888us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:50.040278 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:50.041628 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:50.042618 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:50.042630 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:50.043031 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:50.043811 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:50.043835 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.045076 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.045079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:50.046620 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:50.046634 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.047602 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.047607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:50.048006 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:50.048757 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:50.048782 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:50.052875 (+ 4093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:50.055488 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:50.055496 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:50.055520 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:50.055949 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:50.056302 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:50.056305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:50.056307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:50.056553 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:50.057117 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:50.057125 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:50.057549 (+ 424us) client_negotiation.cc:769] Sending connection context
0927 15:09:50.058755 (+ 1206us) client_negotiation.cc:240] Negotiation successful
0927 15:09:50.059172 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7559,"thread_start_us":173,"threads_started":1}
I20260927 15:09:50.062358 19836 heartbeater.cc:347] Connected to a master server at 127.7.17.254:43727
I20260927 15:09:50.062671 19836 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:50.063395 19836 heartbeater.cc:511] Master 127.7.17.254:43727 requested a full tablet report, sending...
I20260927 15:09:50.064847 19486 ts_manager.cc:194] Registered new tserver with Master: 80a00b28a3794cc99357cd941979bda5 (127.7.17.194:34647)
I20260927 15:09:50.065655 19486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:39571
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
I20260927 15:09:50.084326 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.17.212:43531/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43727
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:09:50 dist-test-slave-hhpq krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521789, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for HTTP/127.7.17.212@KRBTEST.COM
W20260927 15:09:50.250804 19841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:50.251103 19841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:50.251168 19841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:50.255775 19841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:50.255859 19841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:50.255945 19841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:09:50.261857 19841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.17.212:43531/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:09:50.263074 19841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:50.264218 19841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:50.274370 19846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:50.274492 19847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:50.274724 19849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:50.275292 19841 server_base.cc:1034] running on GCE node
I20260927 15:09:50.275724 19841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:50.276436 19841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:50.277611 19841 hybrid_clock.cc:648] HybridClock initialized: now 1790521790277577 us; error 45 us; skew 500 ppm
Sep 27 15:09:50 dist-test-slave-hhpq krb5kdc[19010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521790, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:50.280645 19841 init.cc:377] Logged in from keytab as kudu/127.7.17.195@KRBTEST.COM (short username kudu)
I20260927 15:09:50.281879 19841 webserver.cc:460] Webserver started at http://127.7.17.195:43991/ using document root <none> and password file <none>
I20260927 15:09:50.282631 19841 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:50.282716 19841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:50.282991 19841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:09:50 dist-test-slave-hhpq krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521790, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for HTTP/127.7.17.212@KRBTEST.COM
I20260927 15:09:50.299714 19841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "5109171e12064e2bad34c5c0f7171cc0"
format_stamp: "Formatted at 2026-09-27 15:09:50 on dist-test-slave-hhpq"
server_key: "adaf8db33cdad4a9ae6ae07715113d76"
server_key_iv: "159b911d3d7cea4f6a519d668ee4f778"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:50.301011 19841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "5109171e12064e2bad34c5c0f7171cc0"
format_stamp: "Formatted at 2026-09-27 15:09:50 on dist-test-slave-hhpq"
server_key: "adaf8db33cdad4a9ae6ae07715113d76"
server_key_iv: "159b911d3d7cea4f6a519d668ee4f778"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:50.308362 19841 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.000s
I20260927 15:09:50.324743 19856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:50.326179 19841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:09:50.326310 19841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "5109171e12064e2bad34c5c0f7171cc0"
format_stamp: "Formatted at 2026-09-27 15:09:50 on dist-test-slave-hhpq"
server_key: "adaf8db33cdad4a9ae6ae07715113d76"
server_key_iv: "159b911d3d7cea4f6a519d668ee4f778"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:50.326440 19841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:50.345721 19841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:50.348789 19841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:50.349020 19841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:50.349663 19841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:09:50.350538 19841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:09:50.350585 19841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:50.350636 19841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:09:50.350665 19841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:50.361536 19841 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:41201
I20260927 15:09:50.361563 19969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:41201 every 8 connection(s)
I20260927 15:09:50.362607 19841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:09:50.365258 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 19841
I20260927 15:09:50.365379 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:09:50.365696 7239 external_mini_cluster.cc:1428] Setting key 8785a79916f0fe838440ca5d3f3b175c
Sep 27 15:09:50 dist-test-slave-hhpq krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521790, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:09:50.391753 19702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:50.364562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:49435 (local address 127.7.17.254:43727)
0927 15:09:50.364695 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:50.364700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:50.377583 (+ 12883us) server_negotiation.cc:407] Connection header received
0927 15:09:50.379004 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:50.379008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:50.379070 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:50.379122 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:50.381099 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.381925 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.383763 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.384117 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.384226 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:50.387339 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:50.387344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:50.387348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:50.387392 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:50.389168 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:50.389839 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:50.389843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:50.389845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:50.389950 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:50.390402 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:50.390406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:50.390407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:50.390636 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:50.390711 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:50.391444 (+ 733us) server_negotiation.cc:299] Negotiation successful
0927 15:09:50.391570 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:09:50.393738 19972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:50.365150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:43727 (local address 127.7.17.195:49435)
0927 15:09:50.377301 (+ 12151us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:50.377422 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:50.378713 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:50.379645 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:50.379656 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:50.380078 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:50.380909 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:50.380933 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.382076 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.382080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:50.383615 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:50.383627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.384305 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.384310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:50.384553 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:50.385307 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:50.385330 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:50.387098 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:50.389306 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:50.389313 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:50.389333 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:50.389699 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:50.390066 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:50.390069 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:50.390072 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:50.390287 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:50.390776 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:50.390783 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:50.391144 (+ 361us) client_negotiation.cc:769] Sending connection context
0927 15:09:50.392001 (+ 857us) client_negotiation.cc:240] Negotiation successful
0927 15:09:50.392449 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11800,"thread_start_us":144,"threads_started":1}
I20260927 15:09:50.395182 19970 heartbeater.cc:347] Connected to a master server at 127.7.17.254:43727
I20260927 15:09:50.395493 19970 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:50.396176 19970 heartbeater.cc:511] Master 127.7.17.254:43727 requested a full tablet report, sending...
I20260927 15:09:50.397531 19486 ts_manager.cc:194] Registered new tserver with Master: 5109171e12064e2bad34c5c0f7171cc0 (127.7.17.195:41201)
I20260927 15:09:50.398283 19486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:49435
I20260927 15:09:50.404245 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:09:50.405028 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 19567
I20260927 15:09:50.416720 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 19706
I20260927 15:09:50.423902 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 19841
I20260927 15:09:50.431213 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 19454
I20260927 15:09:50.438072 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:43727
--webserver_interface=127.7.17.254
--webserver_port=36881
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:43727
--ranger_config_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.17.212:43531/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.17.212:43531/kms with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:09:50.548249 19973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:50.548547 19973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:50.548604 19973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:50.552662 19973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:09:50.552743 19973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:50.552769 19973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:50.552790 19973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:09:50.552809 19973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:09:50.558527 19973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.17.212:43531/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:43727
--ranger_config_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:43727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=36881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:09:50.559700 19973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:50.560746 19973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:50.569689 19978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:50.569669 19981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:50.569674 19979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:50.570009 19973 server_base.cc:1034] running on GCE node
I20260927 15:09:50.570600 19973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:50.571807 19973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:50.572962 19973 hybrid_clock.cc:648] HybridClock initialized: now 1790521790572935 us; error 47 us; skew 500 ppm
Sep 27 15:09:50 dist-test-slave-hhpq krb5kdc[19010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521790, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:50.575696 19973 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:09:50.576653 19973 webserver.cc:460] Webserver started at http://127.7.17.254:36881/ using document root <none> and password file <none>
I20260927 15:09:50.577275 19973 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:50.577350 19973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:09:50 dist-test-slave-hhpq krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521790, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for HTTP/127.7.17.212@KRBTEST.COM
I20260927 15:09:50.622043 19973 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:09:50.624588 19989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:50.625989 19973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:09:50.626122 19973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "d8d3311eb7c04023b7256bb40c557a45"
format_stamp: "Formatted at 2026-09-27 15:09:48 on dist-test-slave-hhpq"
server_key: "428700d88bed91b47961df8adc242ef2"
server_key_iv: "4022b313d5513772b390d212704084a8"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:50.626684 19973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:50.642525 19973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:50.645709 19973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:50.645887 19973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:50.653070 19973 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:43727
I20260927 15:09:50.653088 20041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:43727 every 8 connection(s)
I20260927 15:09:50.654201 19973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:09:50.654981 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 19973
I20260927 15:09:50.655365 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.17.212:43531/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:38227
--local_ip_for_outbound_sockets=127.7.17.193
--tserver_master_addrs=127.7.17.254:43727
--webserver_port=34697
--webserver_interface=127.7.17.193
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.17.212:43531/kms with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:09:50.658782 20042 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:09:50.660465 20042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45: Bootstrap starting.
I20260927 15:09:50.676590 20042 log.cc:824] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:50.683476 20042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:09:50.683897 20042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45: Bootstrap complete.
I20260927 15:09:50.688037 20042 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d3311eb7c04023b7256bb40c557a45" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43727 } }
I20260927 15:09:50.688475 20042 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d3311eb7c04023b7256bb40c557a45, State: Initialized, Role: FOLLOWER
I20260927 15:09:50.688987 20042 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d3311eb7c04023b7256bb40c557a45" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43727 } }
I20260927 15:09:50.689116 20042 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:09:50.689183 20042 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:09:50.689276 20042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:09:50.691313 20042 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d3311eb7c04023b7256bb40c557a45" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43727 } }
I20260927 15:09:50.691639 20042 leader_election.cc:302] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8d3311eb7c04023b7256bb40c557a45; no voters:
I20260927 15:09:50.691944 20042 leader_election.cc:288] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:09:50.692049 20046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:09:50.692238 20046 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [term 2 LEADER]: Becoming Leader. State: Replica: d8d3311eb7c04023b7256bb40c557a45, State: Running, Role: LEADER
I20260927 15:09:50.692627 20046 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d3311eb7c04023b7256bb40c557a45" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43727 } }
I20260927 15:09:50.693293 20042 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:09:50.694185 20047 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d8d3311eb7c04023b7256bb40c557a45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d3311eb7c04023b7256bb40c557a45" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43727 } } }
I20260927 15:09:50.694327 20047 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [sys.catalog]: This master's current role is: LEADER
I20260927 15:09:50.694612 20048 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8d3311eb7c04023b7256bb40c557a45. Latest consensus state: current_term: 2 leader_uuid: "d8d3311eb7c04023b7256bb40c557a45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d3311eb7c04023b7256bb40c557a45" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43727 } } }
I20260927 15:09:50.694741 20048 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45 [sys.catalog]: This master's current role is: LEADER
I20260927 15:09:50.695127 20055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:09:50.695703 20042 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 15:09:50.702060 20055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:09:50.703321 20055 catalog_manager.cc:1259] Loaded cluster ID: 14608b39d73244e68691b4f82dea8d4b
I20260927 15:09:50.703544 20055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:09:50.706380 20055 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:09:50.707204 20055 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45: Loaded TSK: 0
I20260927 15:09:50.708658 20055 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:09:50.777357 20044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:50.777626 20044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:50.777686 20044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:50.782212 20044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:50.782284 20044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:50.782368 20044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:09:50.788571 20044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.17.212:43531/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:38227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=34697
--tserver_master_addrs=127.7.17.254:43727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:09:50.789764 20044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:50.791039 20044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:50.801085 20082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:50.801834 20079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:50.802548 20044 server_base.cc:1034] running on GCE node
W20260927 15:09:50.802807 20080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:50.803393 20044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:50.804263 20044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:50.805460 20044 hybrid_clock.cc:648] HybridClock initialized: now 1790521790805437 us; error 44 us; skew 500 ppm
Sep 27 15:09:50 dist-test-slave-hhpq krb5kdc[19010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521790, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:50.810341 20044 init.cc:377] Logged in from keytab as kudu/127.7.17.193@KRBTEST.COM (short username kudu)
I20260927 15:09:50.811616 20044 webserver.cc:460] Webserver started at http://127.7.17.193:34697/ using document root <none> and password file <none>
I20260927 15:09:50.812498 20044 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:50.812614 20044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:09:50 dist-test-slave-hhpq krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521790, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for HTTP/127.7.17.212@KRBTEST.COM
I20260927 15:09:50.838176 20044 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260927 15:09:50.845721 20090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:50.847779 20044 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20260927 15:09:50.847976 20044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "4adb5b1cee8640b5bfc3ae6024c2921c"
format_stamp: "Formatted at 2026-09-27 15:09:49 on dist-test-slave-hhpq"
server_key: "bce4a654995b3c3f8e95573befeb10fa"
server_key_iv: "d80e4477acdb563584bdc3eae088e2c0"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:50.848855 20044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:50.879302 20044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:50.883406 20044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:50.883634 20044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:50.884552 20044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:09:50.885830 20044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:09:50.885919 20044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:50.885982 20044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:09:50.886003 20044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:50.898344 20044 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:38227
I20260927 15:09:50.899653 20203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:38227 every 8 connection(s)
I20260927 15:09:50.899928 20044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:09:50.902082 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 20044
I20260927 15:09:50.902567 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.17.212:43531/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:34647
--local_ip_for_outbound_sockets=127.7.17.194
--tserver_master_addrs=127.7.17.254:43727
--webserver_port=35741
--webserver_interface=127.7.17.194
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.17.212:43531/kms with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:09:50 dist-test-slave-hhpq krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521790, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:09:50.930756 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:50.904598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:39445 (local address 127.7.17.254:43727)
0927 15:09:50.906973 (+ 2375us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:50.907000 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:50.907051 (+ 51us) server_negotiation.cc:407] Connection header received
0927 15:09:50.907387 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:50.907404 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:50.907924 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:50.908364 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:50.910428 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.912066 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.914058 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.914882 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.915067 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:50.918278 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:50.918287 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:50.918310 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:50.918353 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:50.925822 (+ 7469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:50.926712 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:50.926722 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:50.926728 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:50.926926 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:50.927553 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:50.927557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:50.927559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:50.928174 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:50.928342 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:50.929209 (+ 867us) server_negotiation.cc:299] Negotiation successful
0927 15:09:50.929596 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2192,"thread_start_us":157,"threads_started":1}
I20260927 15:09:50.932482 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:50.902786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:43727 (local address 127.7.17.193:39445)
0927 15:09:50.904402 (+ 1616us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:50.904474 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:50.905648 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:50.909044 (+ 3396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:50.909055 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:50.909451 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:50.910190 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:50.910214 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.912270 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.912274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:50.913864 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:50.913884 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:50.915328 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:50.915333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:50.915546 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:50.916246 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:50.916269 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:50.917965 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:50.926016 (+ 8051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:50.926025 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:50.926045 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:50.926530 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:50.927075 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:50.927078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:50.927080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:50.927396 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:50.928411 (+ 1015us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:50.928419 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:50.928773 (+ 354us) client_negotiation.cc:769] Sending connection context
0927 15:09:50.930697 (+ 1924us) client_negotiation.cc:240] Negotiation successful
0927 15:09:50.931428 (+ 731us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1339,"thread_start_us":158,"threads_started":1}
I20260927 15:09:50.935670 20204 heartbeater.cc:347] Connected to a master server at 127.7.17.254:43727
I20260927 15:09:50.935961 20204 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:50.936589 20204 heartbeater.cc:511] Master 127.7.17.254:43727 requested a full tablet report, sending...
W20260927 15:09:50.939292 20204 heartbeater.cc:644] Failed to heartbeat to 127.7.17.254:43727 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 15:09:50.939440 20204 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:50.939636 20204 heartbeater.cc:511] Master 127.7.17.254:43727 requested a full tablet report, sending...
I20260927 15:09:50.940722 20204 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:50.940932 20204 heartbeater.cc:511] Master 127.7.17.254:43727 requested a full tablet report, sending...
W20260927 15:09:50.941774 20204 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:09:51.051542 20207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:51.051856 20207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:51.051920 20207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:51.056397 20207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:51.056483 20207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:51.056567 20207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:09:51.062429 20207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.17.212:43531/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:34647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=35741
--tserver_master_addrs=127.7.17.254:43727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:09:51.063668 20207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:51.064888 20207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:51.076130 20214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:51.076144 20213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:51.076901 20207 server_base.cc:1034] running on GCE node
W20260927 15:09:51.076130 20216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:51.077561 20207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:51.078311 20207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:51.079492 20207 hybrid_clock.cc:648] HybridClock initialized: now 1790521791079456 us; error 50 us; skew 500 ppm
Sep 27 15:09:51 dist-test-slave-hhpq krb5kdc[19010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521791, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:51.082597 20207 init.cc:377] Logged in from keytab as kudu/127.7.17.194@KRBTEST.COM (short username kudu)
I20260927 15:09:51.083652 20207 webserver.cc:460] Webserver started at http://127.7.17.194:35741/ using document root <none> and password file <none>
I20260927 15:09:51.084436 20207 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:51.084556 20207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:09:51 dist-test-slave-hhpq krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521791, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for HTTP/127.7.17.212@KRBTEST.COM
I20260927 15:09:51.108409 20207 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:09:51.111919 20224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:51.113926 20207 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260927 15:09:51.114111 20207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "80a00b28a3794cc99357cd941979bda5"
format_stamp: "Formatted at 2026-09-27 15:09:49 on dist-test-slave-hhpq"
server_key: "34a3a30e25b75a79ef98651af0e8bf48"
server_key_iv: "97dac25247fd08fa521501a815b4ddb5"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:51.114944 20207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:51.136770 20207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:51.140882 20207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:51.141105 20207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:51.141888 20207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:09:51.143074 20207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:09:51.143137 20207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:51.143195 20207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:09:51.143219 20207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:51.157940 20207 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:34647
I20260927 15:09:51.159494 20207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:09:51.160115 20337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:34647 every 8 connection(s)
I20260927 15:09:51.162544 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 20207
I20260927 15:09:51.163007 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.17.212:43531/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:41201
--local_ip_for_outbound_sockets=127.7.17.195
--tserver_master_addrs=127.7.17.254:43727
--webserver_port=43991
--webserver_interface=127.7.17.195
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.17.212:43531/kms with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:09:51 dist-test-slave-hhpq krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521791, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:09:51.179921 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.162528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:47807 (local address 127.7.17.254:43727)
0927 15:09:51.162669 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:51.162673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:51.165466 (+ 2793us) server_negotiation.cc:407] Connection header received
0927 15:09:51.166743 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:51.166747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:51.166879 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:51.166982 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:51.168600 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.169457 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.171314 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.171671 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.171758 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.175082 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:51.175087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:51.175090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:51.175121 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:51.177019 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.177886 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.177890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.177892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.177991 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.178451 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.178454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.178456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.178712 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:51.178788 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:51.179564 (+ 776us) server_negotiation.cc:299] Negotiation successful
0927 15:09:51.179738 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:09:51.182067 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.163010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:43727 (local address 127.7.17.194:47807)
0927 15:09:51.165205 (+ 2195us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.165295 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:51.166439 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:51.167228 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:51.167238 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:51.167621 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:51.168397 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:51.168420 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.169631 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.169634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:51.171127 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:51.171140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.171993 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.171997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:51.172226 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.173022 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:51.173047 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:51.174841 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:51.177204 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:51.177211 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:51.177235 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:51.177706 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:51.178104 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:51.178107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:51.178109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:51.178332 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:51.178876 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:51.178883 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:51.179250 (+ 367us) client_negotiation.cc:769] Sending connection context
0927 15:09:51.180438 (+ 1188us) client_negotiation.cc:240] Negotiation successful
0927 15:09:51.180869 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1935,"thread_start_us":123,"threads_started":1}
I20260927 15:09:51.183861 20338 heartbeater.cc:347] Connected to a master server at 127.7.17.254:43727
I20260927 15:09:51.184185 20338 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:51.184954 20338 heartbeater.cc:511] Master 127.7.17.254:43727 requested a full tablet report, sending...
W20260927 15:09:51.187107 20338 heartbeater.cc:644] Failed to heartbeat to 127.7.17.254:43727 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 15:09:51.187281 20338 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:51.187505 20338 heartbeater.cc:511] Master 127.7.17.254:43727 requested a full tablet report, sending...
I20260927 15:09:51.188624 20338 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:51.188895 20338 heartbeater.cc:511] Master 127.7.17.254:43727 requested a full tablet report, sending...
W20260927 15:09:51.189787 20338 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:09:51.308753 20341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:51.309257 20341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:51.309630 20341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:51.314664 20341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:51.314766 20341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:51.314858 20341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:09:51.322469 20341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.17.212:43531/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:41201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=43991
--tserver_master_addrs=127.7.17.254:43727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:09:51.323632 20341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:51.324658 20341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:51.334772 20348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:51.334772 20350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:51.335264 20341 server_base.cc:1034] running on GCE node
W20260927 15:09:51.336988 20347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:51.337529 20341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:51.338300 20341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:51.341615 20341 hybrid_clock.cc:648] HybridClock initialized: now 1790521791341569 us; error 57 us; skew 500 ppm
Sep 27 15:09:51 dist-test-slave-hhpq krb5kdc[19010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521791, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:51.344993 20341 init.cc:377] Logged in from keytab as kudu/127.7.17.195@KRBTEST.COM (short username kudu)
I20260927 15:09:51.346371 20341 webserver.cc:460] Webserver started at http://127.7.17.195:43991/ using document root <none> and password file <none>
I20260927 15:09:51.347162 20341 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:51.347262 20341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:09:51 dist-test-slave-hhpq krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521791, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for HTTP/127.7.17.212@KRBTEST.COM
I20260927 15:09:51.366679 20341 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:09:51.370558 20358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:51.372437 20341 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:09:51.372573 20341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "5109171e12064e2bad34c5c0f7171cc0"
format_stamp: "Formatted at 2026-09-27 15:09:50 on dist-test-slave-hhpq"
server_key: "adaf8db33cdad4a9ae6ae07715113d76"
server_key_iv: "159b911d3d7cea4f6a519d668ee4f778"
server_key_version: "kuduclusterkey@0"
I20260927 15:09:51.373378 20341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:51.392210 20341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:51.396196 20341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:51.396417 20341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:51.397267 20341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:09:51.398435 20341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:09:51.398504 20341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:51.398558 20341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:09:51.398588 20341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:51.409322 20341 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:41201
I20260927 15:09:51.410642 20341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:09:51.411564 20471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:41201 every 8 connection(s)
I20260927 15:09:51.417009 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 20341
Sep 27 15:09:51 dist-test-slave-hhpq krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521791, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:09:51.429091 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.413998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:45683 (local address 127.7.17.254:43727)
0927 15:09:51.414151 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:51.414156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:51.415174 (+ 1018us) server_negotiation.cc:407] Connection header received
0927 15:09:51.416431 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:51.416436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:51.416503 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:51.416610 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:51.418730 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.419261 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.420917 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.421304 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.421397 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.424609 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:51.424613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:51.424615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:51.424643 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:51.426734 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.427406 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.427409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.427410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.427500 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.427871 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.427874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.427876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.428059 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:51.428143 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:51.428756 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 15:09:51.428923 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:09:51.430193 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.414416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:43727 (local address 127.7.17.195:45683)
0927 15:09:51.414973 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:51.415035 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:51.416144 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:51.417641 (+ 1497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:51.417649 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:51.417960 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:51.418554 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:51.418569 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.419390 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.419394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:51.420703 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:51.420716 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.421632 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.421637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:51.421871 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.422627 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:51.422650 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:51.424419 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:51.426863 (+ 2444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:51.426871 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:51.426892 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:51.427289 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:51.427577 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:51.427581 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:51.427583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:51.427772 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:51.428165 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:51.428173 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:51.428488 (+ 315us) client_negotiation.cc:769] Sending connection context
0927 15:09:51.428700 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 15:09:51.429079 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":140,"threads_started":1}
I20260927 15:09:51.431613 20472 heartbeater.cc:347] Connected to a master server at 127.7.17.254:43727
I20260927 15:09:51.431887 20472 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:51.432493 20472 heartbeater.cc:511] Master 127.7.17.254:43727 requested a full tablet report, sending...
W20260927 15:09:51.434157 20472 heartbeater.cc:644] Failed to heartbeat to 127.7.17.254:43727 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 15:09:51.434271 20472 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:51.434458 20472 heartbeater.cc:511] Master 127.7.17.254:43727 requested a full tablet report, sending...
I20260927 15:09:51.435226 20472 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:51.435391 20472 heartbeater.cc:511] Master 127.7.17.254:43727 requested a full tablet report, sending...
W20260927 15:09:51.436121 20472 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 15:09:51.438545 20476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:51.421159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57778 (local address 127.7.17.254:43727)
0927 15:09:51.423478 (+ 2319us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:51.423482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:51.423493 (+ 11us) server_negotiation.cc:407] Connection header received
0927 15:09:51.423528 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:51.423530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:51.423569 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:51.423647 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:51.424590 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.425752 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.426966 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:51.427289 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:51.427389 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:51.433222 (+ 5833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:51.433226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:51.433229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:51.433260 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:51.436687 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.437254 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.437258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.437260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.437347 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:51.437649 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:51.437652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:51.437654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:51.437845 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:51.438068 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:51.438239 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 15:09:51.438366 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2244,"thread_start_us":99,"threads_started":1}
Sep 27 15:09:51 dist-test-slave-hhpq krb5kdc[19010](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790521791, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:51.942060 20204 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:51.942368 20204 heartbeater.cc:511] Master 127.7.17.254:43727 requested a full tablet report, sending...
I20260927 15:09:51.943691 20006 ts_manager.cc:194] Registered new tserver with Master: 4adb5b1cee8640b5bfc3ae6024c2921c (127.7.17.193:38227)
I20260927 15:09:51.945099 20006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:39445
I20260927 15:09:52.190028 20338 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:52.190361 20338 heartbeater.cc:511] Master 127.7.17.254:43727 requested a full tablet report, sending...
I20260927 15:09:52.191148 20006 ts_manager.cc:194] Registered new tserver with Master: 80a00b28a3794cc99357cd941979bda5 (127.7.17.194:34647)
I20260927 15:09:52.191759 20006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:47807
Sep 27 15:09:52 dist-test-slave-hhpq krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521791, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for HTTP/127.7.17.212@KRBTEST.COM
I20260927 15:09:52.436379 20472 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:52.436712 20472 heartbeater.cc:511] Master 127.7.17.254:43727 requested a full tablet report, sending...
I20260927 15:09:52.437736 20006 ts_manager.cc:194] Registered new tserver with Master: 5109171e12064e2bad34c5c0f7171cc0 (127.7.17.195:41201)
I20260927 15:09:52.438509 20006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:45683
I20260927 15:09:52.442826 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:09:52.458935 20494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:52.444802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57782 (local address 127.7.17.254:43727)
0927 15:09:52.449045 (+ 4243us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:52.449050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:52.449072 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:09:52.449126 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:52.449130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:52.449185 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:52.449306 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:52.450299 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.451173 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.452399 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.452754 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.452895 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:52.454604 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:52.454608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:52.454611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:52.454640 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:52.456469 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:52.457028 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:52.457032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:52.457034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:52.457141 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:52.457489 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:52.457492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:52.457494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:52.457736 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:52.457868 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:52.458593 (+ 725us) server_negotiation.cc:299] Negotiation successful
0927 15:09:52.458741 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4141,"thread_start_us":93,"threads_started":1}
I20260927 15:09:52.463663 20006 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:09:52.468724 20006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:09:52.490670 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:52.483711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:34647 (local address 127.0.0.1:54336)
0927 15:09:52.484271 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:52.484284 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:52.484385 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:52.485462 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:52.485469 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:52.485495 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:09:52.485656 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:52.485664 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.487270 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.487273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:52.488666 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:52.488679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.490088 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.490092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:52.490216 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:52.490278 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 15:09:52.490371 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 15:09:52.490503 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":82,"threads_started":1}
I20260927 15:09:52.491807 20501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:52.483886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54336 (local address 127.7.17.194:34647)
0927 15:09:52.484914 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:52.484921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:52.484947 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:09:52.485028 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:52.485034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:52.485276 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:52.485830 (+ 554us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:09:52.485976 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.487120 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.489089 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.489947 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.491344 (+ 1397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:52.491507 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:52.491562 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 15:09:52.491663 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871,"thread_start_us":93,"threads_started":1}
I20260927 15:09:52.495302 20272 tablet_service.cc:1467] Processing CreateTablet for tablet 0065efa92e144070bf932ea3694ddd31 (DEFAULT_TABLE table=test-table [id=066d020b9f064c739fab3dbd5ac93817]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:09:52.497519 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:52.483987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:41201 (local address 127.0.0.1:43858)
0927 15:09:52.484557 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:52.484568 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:52.484639 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:52.488782 (+ 4143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:52.488785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:52.488804 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:09:52.488931 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:52.488937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.494442 (+ 5505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.494445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:52.495812 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:52.495825 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.496962 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.496965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:52.497058 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:52.497254 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 15:09:52.497302 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 15:09:52.497374 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":159,"thread_start_us":76,"threads_started":1}
I20260927 15:09:52.497567 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:52.484809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43858 (local address 127.7.17.195:41201)
0927 15:09:52.487312 (+ 2503us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:52.487319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:52.487343 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:09:52.487420 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:52.487426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:52.487645 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:52.487831 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:09:52.493341 (+ 5510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.494317 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.495974 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.496786 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.497123 (+ 337us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:52.497299 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:52.497350 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 15:09:52.497441 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2389,"thread_start_us":83,"threads_started":1}
I20260927 15:09:52.497748 20272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0065efa92e144070bf932ea3694ddd31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:52.501166 20393 tablet_service.cc:1467] Processing CreateTablet for tablet 0065efa92e144070bf932ea3694ddd31 (DEFAULT_TABLE table=test-table [id=066d020b9f064c739fab3dbd5ac93817]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:09:52.502336 20393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0065efa92e144070bf932ea3694ddd31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:52.510829 20499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:52.482898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:38227 (local address 127.0.0.1:47146)
0927 15:09:52.483317 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:52.483379 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:52.483517 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:52.497047 (+ 13530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:52.497051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:52.497069 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:09:52.497166 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:52.497172 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.507935 (+ 10763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.507940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:52.509154 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:52.509169 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.510384 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.510388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:52.510491 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:52.510523 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 15:09:52.510580 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 15:09:52.510670 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":103,"threads_started":1}
I20260927 15:09:52.511498 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:52.484488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.7.17.193:38227)
0927 15:09:52.496530 (+ 12042us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:52.496538 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:52.496566 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:09:52.496643 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:52.496648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:52.496910 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:52.498920 (+ 2010us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:09:52.499092 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.500175 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.509354 (+ 9179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.510207 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.511038 (+ 831us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:52.511192 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:52.511260 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 15:09:52.511358 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11923,"thread_start_us":86,"threads_started":1}
I20260927 15:09:52.514891 20122 tablet_service.cc:1467] Processing CreateTablet for tablet 0065efa92e144070bf932ea3694ddd31 (DEFAULT_TABLE table=test-table [id=066d020b9f064c739fab3dbd5ac93817]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:09:52.516012 20506 tablet_bootstrap.cc:492] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0: Bootstrap starting.
I20260927 15:09:52.516108 20122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0065efa92e144070bf932ea3694ddd31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:52.518047 20506 tablet_bootstrap.cc:654] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:52.518563 20507 tablet_bootstrap.cc:492] T 0065efa92e144070bf932ea3694ddd31 P 80a00b28a3794cc99357cd941979bda5: Bootstrap starting.
I20260927 15:09:52.518942 20506 log.cc:824] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:52.521418 20507 tablet_bootstrap.cc:654] T 0065efa92e144070bf932ea3694ddd31 P 80a00b28a3794cc99357cd941979bda5: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:52.522014 20506 tablet_bootstrap.cc:492] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0: No bootstrap required, opened a new log
I20260927 15:09:52.522244 20507 log.cc:824] T 0065efa92e144070bf932ea3694ddd31 P 80a00b28a3794cc99357cd941979bda5: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:52.522301 20506 ts_tablet_manager.cc:1397] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20260927 15:09:52.524552 20507 tablet_bootstrap.cc:492] T 0065efa92e144070bf932ea3694ddd31 P 80a00b28a3794cc99357cd941979bda5: No bootstrap required, opened a new log
I20260927 15:09:52.524739 20507 ts_tablet_manager.cc:1397] T 0065efa92e144070bf932ea3694ddd31 P 80a00b28a3794cc99357cd941979bda5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:09:52.526916 20506 raft_consensus.cc:348] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4adb5b1cee8640b5bfc3ae6024c2921c" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 38227 } } peers { permanent_uuid: "80a00b28a3794cc99357cd941979bda5" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 34647 } } peers { permanent_uuid: "5109171e12064e2bad34c5c0f7171cc0" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41201 } }
I20260927 15:09:52.527220 20506 raft_consensus.cc:374] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:52.527268 20506 raft_consensus.cc:729] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5109171e12064e2bad34c5c0f7171cc0, State: Initialized, Role: FOLLOWER
I20260927 15:09:52.527827 20506 consensus_queue.cc:260] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4adb5b1cee8640b5bfc3ae6024c2921c" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 38227 } } peers { permanent_uuid: "80a00b28a3794cc99357cd941979bda5" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 34647 } } peers { permanent_uuid: "5109171e12064e2bad34c5c0f7171cc0" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41201 } }
I20260927 15:09:52.528909 20506 ts_tablet_manager.cc:1428] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260927 15:09:52.529300 20472 heartbeater.cc:503] Master 127.7.17.254:43727 was elected leader, sending a full tablet report...
I20260927 15:09:52.529556 20510 tablet_bootstrap.cc:492] T 0065efa92e144070bf932ea3694ddd31 P 4adb5b1cee8640b5bfc3ae6024c2921c: Bootstrap starting.
I20260927 15:09:52.531714 20511 raft_consensus.cc:482] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:09:52.531859 20511 raft_consensus.cc:504] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4adb5b1cee8640b5bfc3ae6024c2921c" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 38227 } } peers { permanent_uuid: "80a00b28a3794cc99357cd941979bda5" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 34647 } } peers { permanent_uuid: "5109171e12064e2bad34c5c0f7171cc0" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41201 } }
I20260927 15:09:52.535346 20511 leader_election.cc:288] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4adb5b1cee8640b5bfc3ae6024c2921c (127.7.17.193:38227), 80a00b28a3794cc99357cd941979bda5 (127.7.17.194:34647)
I20260927 15:09:52.528847 20507 raft_consensus.cc:348] T 0065efa92e144070bf932ea3694ddd31 P 80a00b28a3794cc99357cd941979bda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4adb5b1cee8640b5bfc3ae6024c2921c" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 38227 } } peers { permanent_uuid: "80a00b28a3794cc99357cd941979bda5" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 34647 } } peers { permanent_uuid: "5109171e12064e2bad34c5c0f7171cc0" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41201 } }
I20260927 15:09:52.536675 20507 raft_consensus.cc:374] T 0065efa92e144070bf932ea3694ddd31 P 80a00b28a3794cc99357cd941979bda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:52.536757 20507 raft_consensus.cc:729] T 0065efa92e144070bf932ea3694ddd31 P 80a00b28a3794cc99357cd941979bda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a00b28a3794cc99357cd941979bda5, State: Initialized, Role: FOLLOWER
I20260927 15:09:52.537034 20510 tablet_bootstrap.cc:654] T 0065efa92e144070bf932ea3694ddd31 P 4adb5b1cee8640b5bfc3ae6024c2921c: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:52.537281 20507 consensus_queue.cc:260] T 0065efa92e144070bf932ea3694ddd31 P 80a00b28a3794cc99357cd941979bda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4adb5b1cee8640b5bfc3ae6024c2921c" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 38227 } } peers { permanent_uuid: "80a00b28a3794cc99357cd941979bda5" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 34647 } } peers { permanent_uuid: "5109171e12064e2bad34c5c0f7171cc0" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41201 } }
I20260927 15:09:52.537951 20338 heartbeater.cc:503] Master 127.7.17.254:43727 was elected leader, sending a full tablet report...
I20260927 15:09:52.538022 20510 log.cc:824] T 0065efa92e144070bf932ea3694ddd31 P 4adb5b1cee8640b5bfc3ae6024c2921c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:52.538401 20507 ts_tablet_manager.cc:1428] T 0065efa92e144070bf932ea3694ddd31 P 80a00b28a3794cc99357cd941979bda5: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260927 15:09:52.544250 20510 tablet_bootstrap.cc:492] T 0065efa92e144070bf932ea3694ddd31 P 4adb5b1cee8640b5bfc3ae6024c2921c: No bootstrap required, opened a new log
I20260927 15:09:52.544549 20501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:52.535173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:53717 (local address 127.7.17.194:34647)
0927 15:09:52.535300 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:52.535305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:52.539717 (+ 4412us) server_negotiation.cc:407] Connection header received
0927 15:09:52.539843 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:52.539847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:52.539896 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:52.539965 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:09:52.540315 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.541037 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.543262 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.543804 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.543897 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:52.543942 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:52.544344 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 15:09:52.544427 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:09:52.544579 20510 ts_tablet_manager.cc:1397] T 0065efa92e144070bf932ea3694ddd31 P 4adb5b1cee8640b5bfc3ae6024c2921c: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.005s
I20260927 15:09:52.545208 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:52.535041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:34647 (local address 127.7.17.195:53717)
0927 15:09:52.539620 (+ 4579us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:52.539634 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:52.539722 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:52.540049 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:52.540052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:52.540069 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:09:52.540194 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:52.540201 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.541527 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.541530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:52.542632 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:52.542642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.544040 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.544044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:52.544195 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:52.544221 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:09:52.544947 (+ 726us) client_negotiation.cc:240] Negotiation successful
0927 15:09:52.545041 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4513,"mutex_wait_us":1,"thread_start_us":87,"threads_started":1}
I20260927 15:09:52.545464 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:52.534614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:50551 (local address 127.7.17.193:38227)
0927 15:09:52.534732 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:52.534736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:52.538737 (+ 4001us) server_negotiation.cc:407] Connection header received
0927 15:09:52.538957 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:52.538961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:52.539013 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:52.539062 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:09:52.539840 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.540612 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.542144 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.542733 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.542859 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:52.542911 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:52.545267 (+ 2356us) server_negotiation.cc:299] Negotiation successful
0927 15:09:52.545345 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:09:52.546020 20289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0065efa92e144070bf932ea3694ddd31" candidate_uuid: "5109171e12064e2bad34c5c0f7171cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a00b28a3794cc99357cd941979bda5" is_pre_election: true
I20260927 15:09:52.546327 20289 raft_consensus.cc:2454] T 0065efa92e144070bf932ea3694ddd31 P 80a00b28a3794cc99357cd941979bda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5109171e12064e2bad34c5c0f7171cc0 in term 0.
I20260927 15:09:52.546867 20359 leader_election.cc:302] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5109171e12064e2bad34c5c0f7171cc0, 80a00b28a3794cc99357cd941979bda5; no voters:
I20260927 15:09:52.547148 20511 raft_consensus.cc:2781] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:09:52.547264 20511 raft_consensus.cc:482] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:09:52.547317 20511 raft_consensus.cc:3037] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:52.548319 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:52.534231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:38227 (local address 127.7.17.195:50551)
0927 15:09:52.538627 (+ 4396us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:52.538645 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:52.538823 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:52.539208 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:52.539211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:52.539229 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:09:52.539369 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:52.539375 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.540755 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.540758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:52.542014 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:52.542025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.544915 (+ 2890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.544919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:52.545026 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:52.545052 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:09:52.545129 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 15:09:52.545202 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4311,"thread_start_us":121,"threads_started":1}
I20260927 15:09:52.548864 20511 raft_consensus.cc:504] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4adb5b1cee8640b5bfc3ae6024c2921c" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 38227 } } peers { permanent_uuid: "80a00b28a3794cc99357cd941979bda5" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 34647 } } peers { permanent_uuid: "5109171e12064e2bad34c5c0f7171cc0" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41201 } }
I20260927 15:09:52.549042 20154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0065efa92e144070bf932ea3694ddd31" candidate_uuid: "5109171e12064e2bad34c5c0f7171cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4adb5b1cee8640b5bfc3ae6024c2921c" is_pre_election: true
I20260927 15:09:52.549266 20511 leader_election.cc:288] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 [CANDIDATE]: Term 1 election: Requested vote from peers 4adb5b1cee8640b5bfc3ae6024c2921c (127.7.17.193:38227), 80a00b28a3794cc99357cd941979bda5 (127.7.17.194:34647)
I20260927 15:09:52.550282 20154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0065efa92e144070bf932ea3694ddd31" candidate_uuid: "5109171e12064e2bad34c5c0f7171cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4adb5b1cee8640b5bfc3ae6024c2921c"
W20260927 15:09:52.550105 20359 leader_election.cc:341] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4adb5b1cee8640b5bfc3ae6024c2921c (127.7.17.193:38227): Illegal state: must be running to vote when last-logged opid is not known
I20260927 15:09:52.550572 20289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0065efa92e144070bf932ea3694ddd31" candidate_uuid: "5109171e12064e2bad34c5c0f7171cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a00b28a3794cc99357cd941979bda5"
I20260927 15:09:52.550690 20289 raft_consensus.cc:3037] T 0065efa92e144070bf932ea3694ddd31 P 80a00b28a3794cc99357cd941979bda5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:52.550675 20510 raft_consensus.cc:348] T 0065efa92e144070bf932ea3694ddd31 P 4adb5b1cee8640b5bfc3ae6024c2921c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4adb5b1cee8640b5bfc3ae6024c2921c" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 38227 } } peers { permanent_uuid: "80a00b28a3794cc99357cd941979bda5" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 34647 } } peers { permanent_uuid: "5109171e12064e2bad34c5c0f7171cc0" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41201 } }
W20260927 15:09:52.550875 20359 leader_election.cc:341] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4adb5b1cee8640b5bfc3ae6024c2921c (127.7.17.193:38227): Illegal state: must be running to vote when last-logged opid is not known
I20260927 15:09:52.550983 20510 raft_consensus.cc:374] T 0065efa92e144070bf932ea3694ddd31 P 4adb5b1cee8640b5bfc3ae6024c2921c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:52.551048 20510 raft_consensus.cc:729] T 0065efa92e144070bf932ea3694ddd31 P 4adb5b1cee8640b5bfc3ae6024c2921c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4adb5b1cee8640b5bfc3ae6024c2921c, State: Initialized, Role: FOLLOWER
I20260927 15:09:52.551532 20510 consensus_queue.cc:260] T 0065efa92e144070bf932ea3694ddd31 P 4adb5b1cee8640b5bfc3ae6024c2921c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4adb5b1cee8640b5bfc3ae6024c2921c" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 38227 } } peers { permanent_uuid: "80a00b28a3794cc99357cd941979bda5" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 34647 } } peers { permanent_uuid: "5109171e12064e2bad34c5c0f7171cc0" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41201 } }
I20260927 15:09:52.551976 20289 raft_consensus.cc:2454] T 0065efa92e144070bf932ea3694ddd31 P 80a00b28a3794cc99357cd941979bda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5109171e12064e2bad34c5c0f7171cc0 in term 1.
I20260927 15:09:52.552729 20359 leader_election.cc:302] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5109171e12064e2bad34c5c0f7171cc0, 80a00b28a3794cc99357cd941979bda5; no voters: 4adb5b1cee8640b5bfc3ae6024c2921c
I20260927 15:09:52.552800 20510 ts_tablet_manager.cc:1428] T 0065efa92e144070bf932ea3694ddd31 P 4adb5b1cee8640b5bfc3ae6024c2921c: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260927 15:09:52.553027 20511 raft_consensus.cc:2781] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:09:52.553241 20204 heartbeater.cc:503] Master 127.7.17.254:43727 was elected leader, sending a full tablet report...
I20260927 15:09:52.553269 20511 raft_consensus.cc:686] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 [term 1 LEADER]: Becoming Leader. State: Replica: 5109171e12064e2bad34c5c0f7171cc0, State: Running, Role: LEADER
I20260927 15:09:52.553599 20511 consensus_queue.cc:237] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4adb5b1cee8640b5bfc3ae6024c2921c" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 38227 } } peers { permanent_uuid: "80a00b28a3794cc99357cd941979bda5" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 34647 } } peers { permanent_uuid: "5109171e12064e2bad34c5c0f7171cc0" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41201 } }
I20260927 15:09:52.557349 20006 catalog_manager.cc:5495] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5109171e12064e2bad34c5c0f7171cc0 (127.7.17.195). New cstate: current_term: 1 leader_uuid: "5109171e12064e2bad34c5c0f7171cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4adb5b1cee8640b5bfc3ae6024c2921c" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 38227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a00b28a3794cc99357cd941979bda5" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 34647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5109171e12064e2bad34c5c0f7171cc0" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 41201 } health_report { overall_health: HEALTHY } } }
Sep 27 15:09:52 dist-test-slave-hhpq krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521791, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for HTTP/127.7.17.212@KRBTEST.COM
I20260927 15:09:52.583823 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:52.578474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43870 (local address 127.7.17.195:41201)
0927 15:09:52.578663 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:52.578668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:52.578683 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:09:52.578778 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:52.578781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:52.578849 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:52.578956 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:09:52.579395 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.580227 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.581550 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:52.581911 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:52.582004 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:52.582421 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:09:52.583004 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:09:52.583122 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:52.583575 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 15:09:52.583642 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260927 15:09:52.591612 20289 raft_consensus.cc:1261] T 0065efa92e144070bf932ea3694ddd31 P 80a00b28a3794cc99357cd941979bda5 [term 1 FOLLOWER]: Refusing update from remote peer 5109171e12064e2bad34c5c0f7171cc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:09:52.592548 20511 consensus_queue.cc:1035] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a00b28a3794cc99357cd941979bda5" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:09:52.601218 20154 raft_consensus.cc:3037] T 0065efa92e144070bf932ea3694ddd31 P 4adb5b1cee8640b5bfc3ae6024c2921c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:52.602855 20154 raft_consensus.cc:1261] T 0065efa92e144070bf932ea3694ddd31 P 4adb5b1cee8640b5bfc3ae6024c2921c [term 1 FOLLOWER]: Refusing update from remote peer 5109171e12064e2bad34c5c0f7171cc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:09:52.603746 20511 consensus_queue.cc:1035] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4adb5b1cee8640b5bfc3ae6024c2921c" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 38227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:09:52.617762 20520 mvcc.cc:205] Tried to move back new op lower bound from 7333977262447136768 to 7333977262304227328. Current Snapshot: MvccSnapshot[applied={T|T < 7333977262447136768}]
I20260927 15:09:52.621728 20523 mvcc.cc:205] Tried to move back new op lower bound from 7333977262447136768 to 7333977262304227328. Current Snapshot: MvccSnapshot[applied={T|T < 7333977262447136768}]
I20260927 15:09:52.631798 20005 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:09:52.631994 20005 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:09:52.635063 20005 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8d3311eb7c04023b7256bb40c557a45: Sending DeleteTablet for 3 replicas of tablet 0065efa92e144070bf932ea3694ddd31
I20260927 15:09:52.636418 20272 tablet_service.cc:1514] Processing DeleteTablet for tablet 0065efa92e144070bf932ea3694ddd31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:09:52 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:54336
I20260927 15:09:52.636471 20122 tablet_service.cc:1514] Processing DeleteTablet for tablet 0065efa92e144070bf932ea3694ddd31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:09:52 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:47146
I20260927 15:09:52.637233 20393 tablet_service.cc:1514] Processing DeleteTablet for tablet 0065efa92e144070bf932ea3694ddd31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:09:52 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:43858
I20260927 15:09:52.638864 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 20044
I20260927 15:09:52.641870 20530 tablet_replica.cc:330] T 0065efa92e144070bf932ea3694ddd31 P 80a00b28a3794cc99357cd941979bda5: stopping tablet replica
I20260927 15:09:52.642367 20530 raft_consensus.cc:2229] T 0065efa92e144070bf932ea3694ddd31 P 80a00b28a3794cc99357cd941979bda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:09:52.642940 20530 raft_consensus.cc:2258] T 0065efa92e144070bf932ea3694ddd31 P 80a00b28a3794cc99357cd941979bda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:09:52.648667 20532 tablet_replica.cc:330] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0: stopping tablet replica
I20260927 15:09:52.649170 20532 raft_consensus.cc:2229] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:09:52.649662 20532 raft_consensus.cc:2258] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:09:52.650564 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 20207
W20260927 15:09:52.651227 19990 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.17.193:38227 (error 108)
I20260927 15:09:52.653241 20499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:52.652939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:38227 (local address 127.0.0.1:47152)
0927 15:09:52.653076 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:52.653155 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.193:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 15:09:52.653440 19990 catalog_manager.cc:4584] TS 4adb5b1cee8640b5bfc3ae6024c2921c (127.7.17.193:38227): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0065efa92e144070bf932ea3694ddd31: Network error: Client connection negotiation failed: client connection to 127.7.17.193:38227: connect: Connection refused (error 111)
I20260927 15:09:52.653762 20532 ts_tablet_manager.cc:1905] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:09:52.656567 20532 ts_tablet_manager.cc:1918] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:09:52.656673 20532 log.cc:1196] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/0065efa92e144070bf932ea3694ddd31
I20260927 15:09:52.657024 20532 ts_tablet_manager.cc:1939] T 0065efa92e144070bf932ea3694ddd31 P 5109171e12064e2bad34c5c0f7171cc0: Deleting consensus metadata
I20260927 15:09:52.658366 19990 catalog_manager.cc:4841] TS 5109171e12064e2bad34c5c0f7171cc0 (127.7.17.195:41201): tablet 0065efa92e144070bf932ea3694ddd31 (table test-table [id=066d020b9f064c739fab3dbd5ac93817]) successfully deleted
I20260927 15:09:52.663786 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 20341
I20260927 15:09:52.665007 20499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:52.664627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:34647 (local address 127.0.0.1:54350)
0927 15:09:52.664855 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:52.664917 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.194:34647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 15:09:52.674081 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 19973
I20260927 15:09:52.681392 7239 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 15:09:53.026436 7239 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 15:09:53.026571 7239 mini_ranger.cc:66] Stopping Ranger...
I20260927 15:09:53.361967 7239 mini_ranger.cc:68] Stopped Ranger
2026-09-27 15:09:53.362 UTC [19040] LOG: received smart shutdown request
2026-09-27 15:09:53.366 UTC [19040] LOG: background worker "logical replication launcher" (PID 19049) exited with exit code 1
2026-09-27 15:09:53.366 UTC [19044] LOG: shutting down
2026-09-27 15:09:53.538 UTC [19040] LOG: database system is shut down
2026-09-27T15:09:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49240 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:09:53 dist-test-slave-hhpq krb5kdc[20542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:09:53 dist-test-slave-hhpq krb5kdc[20542](info): set up 2 sockets
Sep 27 15:09:53 dist-test-slave-hhpq krb5kdc[20542](info): commencing operation
krb5kdc: starting...
W20260927 15:09:54.600801 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:09:54 dist-test-slave-hhpq krb5kdc[20542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:09:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:09:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:09:54.724741 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:41865
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:41865,127.7.17.253:38055,127.7.17.252:39549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:09:54.836586 20555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:54.836967 20555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:54.837044 20555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:54.841225 20555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:09:54.841307 20555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:54.841336 20555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:54.841358 20555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:09:54.841377 20555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:09:54.847087 20555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:41865,127.7.17.253:38055,127.7.17.252:39549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:41865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:09:54.848312 20555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:54.849437 20555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:54.858191 20561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:54.858228 20555 server_base.cc:1034] running on GCE node
W20260927 15:09:54.858233 20560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:54.858246 20563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:54.859058 20555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:54.860317 20555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:54.861475 20555 hybrid_clock.cc:648] HybridClock initialized: now 1790521794861453 us; error 40 us; skew 500 ppm
Sep 27 15:09:54 dist-test-slave-hhpq krb5kdc[20542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521794, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:54.864342 20555 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:09:54.865411 20555 webserver.cc:460] Webserver started at http://127.7.17.254:41337/ using document root <none> and password file <none>
I20260927 15:09:54.865998 20555 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:54.866062 20555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:54.866273 20555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:09:54.868433 20555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb"
format_stamp: "Formatted at 2026-09-27 15:09:54 on dist-test-slave-hhpq"
server_key: "64aa3b584d155205f33a82079633df3f"
server_key_iv: "1ee9cb9ef1d944b56fd025a655778d33"
server_key_version: "encryptionkey@0"
I20260927 15:09:54.869267 20555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb"
format_stamp: "Formatted at 2026-09-27 15:09:54 on dist-test-slave-hhpq"
server_key: "64aa3b584d155205f33a82079633df3f"
server_key_iv: "1ee9cb9ef1d944b56fd025a655778d33"
server_key_version: "encryptionkey@0"
I20260927 15:09:54.873376 20555 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:09:54.875917 20570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:54.877316 20555 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:09:54.877445 20555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb"
format_stamp: "Formatted at 2026-09-27 15:09:54 on dist-test-slave-hhpq"
server_key: "64aa3b584d155205f33a82079633df3f"
server_key_iv: "1ee9cb9ef1d944b56fd025a655778d33"
server_key_version: "encryptionkey@0"
I20260927 15:09:54.877568 20555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:54.887229 20555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:54.890507 20555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:54.890703 20555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:54.897908 20555 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:41865
I20260927 15:09:54.897933 20622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:41865 every 8 connection(s)
I20260927 15:09:54.898939 20555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:09:54.901095 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 20555
I20260927 15:09:54.901259 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:09:54.901582 7239 external_mini_cluster.cc:1428] Setting key 4e801172673f782fd910a82dbc19f515
I20260927 15:09:54.902326 20623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:54.906708 20623 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41865 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.17.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.253@KRBTEST.COM" created.
Sep 27 15:09:54 dist-test-slave-hhpq krb5kdc[20542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521794, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:09:54.922937 20627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:54.909343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.7.17.254:41865)
0927 15:09:54.909810 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:54.909829 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:54.910126 (+ 297us) server_negotiation.cc:407] Connection header received
0927 15:09:54.911130 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:54.911136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:54.911385 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:54.911548 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:54.912504 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:54.913473 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:54.914564 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:54.914968 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:54.915097 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:54.918029 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:54.918035 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:54.918049 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:54.918074 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:54.919699 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:54.920229 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:54.920234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:54.920235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:54.920343 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:54.920656 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:54.920659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:54.920660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:54.921119 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:54.921230 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:54.921750 (+ 520us) server_negotiation.cc:299] Negotiation successful
0927 15:09:54.922093 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":148,"server-negotiator.queue_time_us":336,"spinlock_wait_cycles":24448,"thread_start_us":162,"threads_started":1}
I20260927 15:09:54.922943 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:54.909344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:41865 (local address 127.0.0.1:60566)
0927 15:09:54.909845 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:54.909887 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:54.910750 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:54.911575 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:54.911582 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:54.911988 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:54.912318 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:54.912338 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:54.913600 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:54.913602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:54.914436 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:54.914448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:54.915128 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:54.915131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:54.915252 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:54.915902 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:54.915928 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:54.917839 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:54.919847 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:54.919861 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:54.919871 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:54.920131 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:54.920437 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:54.920440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:54.920443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:54.920583 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:54.921252 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:54.921260 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:54.921439 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 15:09:54.921628 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 15:09:54.922000 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":3,"thread_start_us":125,"threads_started":1}
I20260927 15:09:54.925670 20623 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:09:54.926578 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:54.926250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:38055 (local address 127.0.0.1:34530)
0927 15:09:54.926413 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:54.926498 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.253:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260927 15:09:54.926873 20571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.17.253:38055: connect: Connection refused (error 111)
I20260927 15:09:54.927803 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:54.927586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:38055 (local address 127.0.0.1:34536)
0927 15:09:54.927671 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:54.927733 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.253:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 15:09:54.927958 20623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.253:38055: Network error: Client connection negotiation failed: client connection to 127.7.17.253:38055: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.253.keytab.
Entry for principal kudu/127.7.17.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.253.keytab.
I20260927 15:09:54.929358 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.253:38055
--webserver_interface=127.7.17.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.253.keytab
--principal=kudu/127.7.17.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:41865,127.7.17.253:38055,127.7.17.252:39549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:09:54.977236 20623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } attempt: 1
I20260927 15:09:54.978111 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:54.977799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:38055 (local address 127.0.0.1:34542)
0927 15:09:54.977968 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:54.978032 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.253:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 15:09:54.978998 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:54.978692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:38055 (local address 127.0.0.1:34544)
0927 15:09:54.978881 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:54.978934 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.253:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260927 15:09:54.979135 20623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.253:38055: Network error: Client connection negotiation failed: client connection to 127.7.17.253:38055: connect: Connection refused (error 111)
W20260927 15:09:55.042783 20630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:55.043092 20630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:55.043149 20630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:55.047250 20630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:09:55.047322 20630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:55.047350 20630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:55.047371 20630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:09:55.047391 20630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:09:55.047297 20623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } attempt: 2
I20260927 15:09:55.048164 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.047850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:38055 (local address 127.0.0.1:34550)
0927 15:09:55.048014 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:55.048078 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.253:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 15:09:55.048913 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.048652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:38055 (local address 127.0.0.1:34564)
0927 15:09:55.048759 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:55.048806 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.253:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 15:09:55.049057 20623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.253:38055: Network error: Client connection negotiation failed: client connection to 127.7.17.253:38055: connect: Connection refused (error 111)
I20260927 15:09:55.053243 20630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:41865,127.7.17.253:38055,127.7.17.252:39549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.253.keytab
--principal=kudu/127.7.17.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.253:38055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.17.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:09:55.054533 20630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:55.055583 20630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:55.064535 20635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:55.064564 20630 server_base.cc:1034] running on GCE node
W20260927 15:09:55.064512 20636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:55.064512 20638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:55.065434 20630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:55.066574 20630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:55.067754 20630 hybrid_clock.cc:648] HybridClock initialized: now 1790521795067730 us; error 36 us; skew 500 ppm
Sep 27 15:09:55 dist-test-slave-hhpq krb5kdc[20542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521795, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:55.070749 20630 init.cc:377] Logged in from keytab as kudu/127.7.17.253@KRBTEST.COM (short username kudu)
I20260927 15:09:55.071744 20630 webserver.cc:460] Webserver started at http://127.7.17.253:37811/ using document root <none> and password file <none>
I20260927 15:09:55.072324 20630 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:55.072419 20630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:55.072635 20630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:09:55.074826 20630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-1/data/instance:
uuid: "53ebbb8495c54140b19f134bd1cb05c2"
format_stamp: "Formatted at 2026-09-27 15:09:55 on dist-test-slave-hhpq"
server_key: "4194f580848fd4215b7520ada9f49c05"
server_key_iv: "b7a024f7a66d28598f1ed1b3f09284c5"
server_key_version: "encryptionkey@0"
I20260927 15:09:55.075457 20630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-1/wal/instance:
uuid: "53ebbb8495c54140b19f134bd1cb05c2"
format_stamp: "Formatted at 2026-09-27 15:09:55 on dist-test-slave-hhpq"
server_key: "4194f580848fd4215b7520ada9f49c05"
server_key_iv: "b7a024f7a66d28598f1ed1b3f09284c5"
server_key_version: "encryptionkey@0"
I20260927 15:09:55.079406 20630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:09:55.082273 20645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:55.083624 20630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:09:55.083768 20630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-1/wal
uuid: "53ebbb8495c54140b19f134bd1cb05c2"
format_stamp: "Formatted at 2026-09-27 15:09:55 on dist-test-slave-hhpq"
server_key: "4194f580848fd4215b7520ada9f49c05"
server_key_iv: "b7a024f7a66d28598f1ed1b3f09284c5"
server_key_version: "encryptionkey@0"
I20260927 15:09:55.083949 20630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:55.094966 20630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:55.098058 20630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:55.098233 20630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:55.105225 20630 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.253:38055
I20260927 15:09:55.105247 20697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.253:38055 every 8 connection(s)
I20260927 15:09:55.106280 20630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-1/data/info.pb
I20260927 15:09:55.109139 20698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:55.113005 20698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41865 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:09:55.115888 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 20630
I20260927 15:09:55.116036 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-1/wal/instance
I20260927 15:09:55.116344 7239 external_mini_cluster.cc:1428] Setting key 6bbedfaaaea5fe0b715f0a8783deb62f
Sep 27 15:09:55 dist-test-slave-hhpq krb5kdc[20542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521795, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.253@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.17.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.252@KRBTEST.COM" created.
I20260927 15:09:55.128307 20627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.114992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60576 (local address 127.7.17.254:41865)
0927 15:09:55.115189 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:55.115193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:55.116064 (+ 871us) server_negotiation.cc:407] Connection header received
0927 15:09:55.117413 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:55.117416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:55.117469 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:55.117542 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:55.119108 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.119819 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.121105 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.121345 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.121436 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:55.124185 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:55.124188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:55.124190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:55.124217 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:55.125971 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.126540 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.126544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.126546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.126639 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.127056 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.127058 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.127060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.127238 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:55.127307 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:55.128006 (+ 699us) server_negotiation.cc:299] Negotiation successful
0927 15:09:55.128135 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 15:09:55.129470 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.115302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:41865 (local address 127.0.0.1:60576)
0927 15:09:55.115878 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:55.115944 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:55.117130 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:55.117749 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:55.117758 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:55.118203 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:55.118931 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:55.118966 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.119957 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.119960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:55.120991 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:55.121001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.121498 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.121502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:55.121684 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:55.122369 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:55.122387 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:55.124012 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:55.126094 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:55.126099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:55.126113 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:55.126443 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:55.126755 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:55.126759 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:55.126761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:55.126967 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:55.127324 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:55.127331 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:55.127671 (+ 340us) client_negotiation.cc:769] Sending connection context
0927 15:09:55.127927 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 15:09:55.128338 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":116,"threads_started":1}
I20260927 15:09:55.130884 20698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:09:55 dist-test-slave-hhpq krb5kdc[20542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521795, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.253@KRBTEST.COM for kudu/127.7.17.253@KRBTEST.COM
I20260927 15:09:55.140223 20623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } attempt: 3
I20260927 15:09:55.141759 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.131417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:38055 (local address 127.0.0.1:34570)
0927 15:09:55.131602 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:55.131616 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:55.131697 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:55.132165 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:55.132168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:55.132352 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:55.132466 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:55.132473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.133556 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.133560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:55.134430 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:55.134442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.135134 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.135138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:55.135214 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:55.135676 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:55.135689 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:55.137705 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:55.139928 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:55.139931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:55.139932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:55.140176 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:55.140525 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:55.140528 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:55.140530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:55.140635 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:55.141249 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:55.141252 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:55.141352 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 15:09:55.141445 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 15:09:55.141582 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260927 15:09:55.141870 20703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.131524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34570 (local address 127.7.17.253:38055)
0927 15:09:55.131810 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:55.131822 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:55.131841 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:09:55.131876 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:55.131881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:55.132042 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:55.132153 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:55.132583 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.133435 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.134561 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.135024 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.135114 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:55.137915 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:55.137927 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:55.137941 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:55.137965 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:55.139733 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.140296 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.140303 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.140305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.140409 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.140756 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.140759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.140761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.141157 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:55.141294 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:55.141538 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 15:09:55.141676 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":82,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.252.keytab.
Entry for principal kudu/127.7.17.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.252.keytab.
I20260927 15:09:55.143030 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.252:39549
--webserver_interface=127.7.17.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.252.keytab
--principal=kudu/127.7.17.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:41865,127.7.17.253:38055,127.7.17.252:39549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:09:55.143895 20698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:09:55.144708 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.144379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:39549 (local address 127.0.0.1:47922)
0927 15:09:55.144552 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:55.144629 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:39549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260927 15:09:55.144912 20647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.17.252:39549: connect: Connection refused (error 111)
I20260927 15:09:55.145921 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.145543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:39549 (local address 127.0.0.1:47936)
0927 15:09:55.145798 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:55.145846 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:39549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":92,"threads_started":1}
W20260927 15:09:55.146160 20698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.252:39549: Network error: Client connection negotiation failed: client connection to 127.7.17.252:39549: connect: Connection refused (error 111)
Sep 27 15:09:55 dist-test-slave-hhpq krb5kdc[20542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521794, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for kudu/127.7.17.253@KRBTEST.COM
I20260927 15:09:55.150251 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.140820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:38055 (local address 127.0.0.1:34580)
0927 15:09:55.140964 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:55.140981 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:55.141064 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:55.141526 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:55.141529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:55.141678 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:55.141782 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:55.141788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.143119 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.143123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:55.143906 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:55.143919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.145120 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.145124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:55.145238 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:55.145718 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:55.145733 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:55.147031 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:55.148924 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:55.148927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:55.148930 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:55.149128 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:55.149399 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:55.149400 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:55.149402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:55.149467 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:55.149872 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:55.149874 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:55.149957 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 15:09:55.150026 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 15:09:55.150133 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 15:09:55.150381 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.140925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34580 (local address 127.7.17.253:38055)
0927 15:09:55.141305 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:55.141311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:55.141320 (+ 9us) server_negotiation.cc:407] Connection header received
0927 15:09:55.141352 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:55.141354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:55.141389 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:55.141465 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:55.142398 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.142995 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.144027 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.144965 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.145102 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:55.147179 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:55.147183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:55.147186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:55.147213 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:55.148729 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.149214 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.149217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.149219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.149310 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.149540 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.149543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.149545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.149788 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:55.149884 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:55.150095 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 15:09:55.150212 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":187,"threads_started":1}
I20260927 15:09:55.151261 20623 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:09:55.151980 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.151742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:39549 (local address 127.0.0.1:47950)
0927 15:09:55.151855 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:55.151908 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:39549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 15:09:55.152673 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.152440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:39549 (local address 127.0.0.1:47964)
0927 15:09:55.152584 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:55.152627 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:39549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260927 15:09:55.152820 20623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.252:39549: Network error: Client connection negotiation failed: client connection to 127.7.17.252:39549: connect: Connection refused (error 111)
I20260927 15:09:55.184032 20623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } attempt: 1
I20260927 15:09:55.184960 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.184602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:39549 (local address 127.0.0.1:47970)
0927 15:09:55.184762 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:55.184870 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:39549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 15:09:55.185705 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.185493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:39549 (local address 127.0.0.1:47984)
0927 15:09:55.185599 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:55.185646 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:39549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 15:09:55.185837 20623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.252:39549: Network error: Client connection negotiation failed: client connection to 127.7.17.252:39549: connect: Connection refused (error 111)
I20260927 15:09:55.195442 20698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } attempt: 1
I20260927 15:09:55.196346 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.195964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:39549 (local address 127.0.0.1:47988)
0927 15:09:55.196136 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:55.196221 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:39549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 15:09:55.197153 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.196922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:39549 (local address 127.0.0.1:47996)
0927 15:09:55.197024 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:55.197084 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:39549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 15:09:55.197302 20698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.252:39549: Network error: Client connection negotiation failed: client connection to 127.7.17.252:39549: connect: Connection refused (error 111)
W20260927 15:09:55.256728 20705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:55.257076 20705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:55.257121 20705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 15:09:55.261018 20623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } attempt: 2
W20260927 15:09:55.261360 20705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:09:55.261418 20705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:55.261437 20705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:55.261454 20705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:09:55.261482 20705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:09:55.261883 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.261585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:39549 (local address 127.0.0.1:48002)
0927 15:09:55.261729 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:55.261793 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:39549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 15:09:55.262617 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.262390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:39549 (local address 127.0.0.1:48004)
0927 15:09:55.262512 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:55.262559 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:39549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 15:09:55.262728 20623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.252:39549: Network error: Client connection negotiation failed: client connection to 127.7.17.252:39549: connect: Connection refused (error 111)
I20260927 15:09:55.265450 20698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } attempt: 2
I20260927 15:09:55.266201 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.265927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:39549 (local address 127.0.0.1:48018)
0927 15:09:55.266057 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:55.266121 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:39549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 15:09:55.266849 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.266659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:39549 (local address 127.0.0.1:48032)
0927 15:09:55.266740 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:55.266785 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:39549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 15:09:55.266961 20698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.252:39549: Network error: Client connection negotiation failed: client connection to 127.7.17.252:39549: connect: Connection refused (error 111)
I20260927 15:09:55.267657 20705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:41865,127.7.17.253:38055,127.7.17.252:39549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.252.keytab
--principal=kudu/127.7.17.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.252:39549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.17.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:09:55.268772 20705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:55.269841 20705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:55.278752 20712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:55.278772 20715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:55.278744 20713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:55.279151 20705 server_base.cc:1034] running on GCE node
I20260927 15:09:55.279771 20705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:55.281087 20705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:55.282332 20705 hybrid_clock.cc:648] HybridClock initialized: now 1790521795282301 us; error 49 us; skew 500 ppm
Sep 27 15:09:55 dist-test-slave-hhpq krb5kdc[20542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521795, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:55.285351 20705 init.cc:377] Logged in from keytab as kudu/127.7.17.252@KRBTEST.COM (short username kudu)
I20260927 15:09:55.286319 20705 webserver.cc:460] Webserver started at http://127.7.17.252:41445/ using document root <none> and password file <none>
I20260927 15:09:55.286897 20705 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:55.286967 20705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:55.287220 20705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:09:55.289572 20705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-2/data/instance:
uuid: "9f2dde15ca784aa6a31b7fbff2cb759e"
format_stamp: "Formatted at 2026-09-27 15:09:55 on dist-test-slave-hhpq"
server_key: "f1f651fb17e4c0ba91b30fddf36d092f"
server_key_iv: "e4bbf5409da8065c4275a5f105428ca2"
server_key_version: "encryptionkey@0"
I20260927 15:09:55.290148 20705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-2/wal/instance:
uuid: "9f2dde15ca784aa6a31b7fbff2cb759e"
format_stamp: "Formatted at 2026-09-27 15:09:55 on dist-test-slave-hhpq"
server_key: "f1f651fb17e4c0ba91b30fddf36d092f"
server_key_iv: "e4bbf5409da8065c4275a5f105428ca2"
server_key_version: "encryptionkey@0"
I20260927 15:09:55.294087 20705 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 15:09:55.296772 20722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:55.298241 20705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:09:55.298396 20705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-2/wal
uuid: "9f2dde15ca784aa6a31b7fbff2cb759e"
format_stamp: "Formatted at 2026-09-27 15:09:55 on dist-test-slave-hhpq"
server_key: "f1f651fb17e4c0ba91b30fddf36d092f"
server_key_iv: "e4bbf5409da8065c4275a5f105428ca2"
server_key_version: "encryptionkey@0"
I20260927 15:09:55.298522 20705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:55.322758 20705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:55.326179 20705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:55.326378 20705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:55.333081 20705 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.252:39549
I20260927 15:09:55.333107 20774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.252:39549 every 8 connection(s)
I20260927 15:09:55.334064 20705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-2/data/info.pb
I20260927 15:09:55.337026 20775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:55.340037 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 20705
I20260927 15:09:55.340155 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521636012660-7239-0/minicluster-data/master-2/wal/instance
I20260927 15:09:55.340415 7239 external_mini_cluster.cc:1428] Setting key dbdc7bd13dceea90bb9925f7d9472305
I20260927 15:09:55.341020 20775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41865 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:09:55 dist-test-slave-hhpq krb5kdc[20542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521795, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.252@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
Sep 27 15:09:55 dist-test-slave-hhpq krb5kdc[20542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.252@KRBTEST.COM
I20260927 15:09:55.353220 20627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.343051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.7.17.254:41865)
0927 15:09:55.343161 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:55.343166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:55.343505 (+ 339us) server_negotiation.cc:407] Connection header received
0927 15:09:55.343775 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:55.343779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:55.343834 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:55.343914 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:55.344738 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.345279 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.346290 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.346513 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.346601 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:55.349175 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:55.349179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:55.349182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:55.349212 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:55.350786 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.351405 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.351408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.351409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.351482 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.351810 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.351813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.351815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.352105 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:55.352204 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:55.352928 (+ 724us) server_negotiation.cc:299] Negotiation successful
0927 15:09:55.353058 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:09:55.354421 20779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.342996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:41865 (local address 127.0.0.1:60584)
0927 15:09:55.343367 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:55.343409 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:55.343599 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:55.343981 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:55.343986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:55.344328 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:55.344594 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:55.344606 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.345378 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.345382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:55.346174 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:55.346183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.346656 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.346659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:55.346826 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:55.347391 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:55.347406 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:55.349007 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:55.350938 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:55.350946 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:55.350959 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:55.351312 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:55.351572 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:55.351575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:55.351577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:55.351724 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:55.352230 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:55.352238 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:55.352626 (+ 388us) client_negotiation.cc:769] Sending connection context
0927 15:09:55.352802 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 15:09:55.353421 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":266,"thread_start_us":137,"threads_started":1}
I20260927 15:09:55.354434 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.341845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48038 (local address 127.7.17.252:39549)
0927 15:09:55.342309 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:55.342323 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:55.342361 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:09:55.343174 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:55.343198 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:55.343610 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:55.343900 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:55.344706 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.345617 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.346465 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.346840 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.346929 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:55.349665 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:55.349672 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:55.349687 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:55.349713 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:55.351467 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.351868 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.351873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.351874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.351938 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.352224 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.352227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.352229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.352606 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:55.352811 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:55.353017 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 15:09:55.353176 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":145,"threads_started":1}
I20260927 15:09:55.355717 20775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:09:55.358098 20698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } attempt: 3
Sep 27 15:09:55 dist-test-slave-hhpq krb5kdc[20542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521795, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.252@KRBTEST.COM for kudu/127.7.17.253@KRBTEST.COM
Sep 27 15:09:55 dist-test-slave-hhpq krb5kdc[20542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:09:55.361866 20623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } attempt: 3
Sep 27 15:09:55 dist-test-slave-hhpq krb5kdc[20542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521795, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.253@KRBTEST.COM for kudu/127.7.17.252@KRBTEST.COM
I20260927 15:09:55.365556 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.356412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34592 (local address 127.7.17.253:38055)
0927 15:09:55.356535 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:55.356538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:55.356551 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:09:55.356587 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:55.356590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:55.356636 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:55.356712 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:55.357189 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.357781 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.358722 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.359089 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.359203 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:55.361315 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:55.361320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:55.361322 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:55.361361 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:55.363270 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.363756 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.363760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.363762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.363873 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.364116 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.364120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.364121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.364566 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:55.364649 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:55.365256 (+ 607us) server_negotiation.cc:299] Negotiation successful
0927 15:09:55.365386 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":54,"spinlock_wait_cycles":34048}
I20260927 15:09:55.366266 20779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.356187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:38055 (local address 127.0.0.1:34592)
0927 15:09:55.356319 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:55.356334 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:55.356440 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:55.356745 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:55.356747 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:55.356940 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:55.357069 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:55.357076 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.357894 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.357896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:55.358612 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:55.358622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.359197 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.359200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:55.359295 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:55.359768 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:55.359783 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:55.361178 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:55.363398 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:55.363401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:55.363403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:55.363640 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:55.363957 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:55.363960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:55.363962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:55.364039 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:55.364692 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:55.364695 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:55.365069 (+ 374us) client_negotiation.cc:769] Sending connection context
0927 15:09:55.365979 (+ 910us) client_negotiation.cc:240] Negotiation successful
0927 15:09:55.366117 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 15:09:55.366366 20627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.356993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60586 (local address 127.7.17.254:41865)
0927 15:09:55.357103 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:55.357106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:55.357118 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:09:55.357175 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:55.357177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:55.357217 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:55.357293 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:55.357844 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.358443 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.359486 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.359801 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.359903 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:55.362551 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:55.362554 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:55.362557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:55.362583 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:55.364183 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.364646 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.364649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.364651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.364738 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.365313 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.365316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.365318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.365533 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:55.365741 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:55.366117 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 15:09:55.366221 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:09:55.367120 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.358485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:39549 (local address 127.0.0.1:48054)
0927 15:09:55.358623 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:55.358637 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:55.358718 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:55.358937 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:55.358939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:55.359105 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:55.359275 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:55.359283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.360242 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.360245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:55.361155 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:55.361167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.362053 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.362057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:55.362160 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:55.362514 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:55.362530 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:55.363870 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:55.365682 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:55.365686 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:55.365688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:55.365916 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:55.366196 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:55.366199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:55.366201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:55.366276 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:55.366660 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:55.366663 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:55.366752 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 15:09:55.366827 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 15:09:55.366954 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":22}
I20260927 15:09:55.367316 20775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:09:55.367887 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.358538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48054 (local address 127.7.17.252:39549)
0927 15:09:55.358776 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:55.358780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:55.358790 (+ 10us) server_negotiation.cc:407] Connection header received
0927 15:09:55.358817 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:55.358819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:55.358852 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:55.358908 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:55.359387 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.360123 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.361597 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.361885 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.362355 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:55.364033 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:55.364038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:55.364040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:55.364066 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:55.365558 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.366031 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.366034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.366036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.366108 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.366378 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.366381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.366382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.366564 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:55.367575 (+ 1011us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:55.367643 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 15:09:55.367763 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
Sep 27 15:09:55 dist-test-slave-hhpq krb5kdc[20542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521794, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for kudu/127.7.17.252@KRBTEST.COM
I20260927 15:09:55.371552 20698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53ebbb8495c54140b19f134bd1cb05c2: Bootstrap starting.
Sep 27 15:09:55 dist-test-slave-hhpq krb5kdc[20542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521795, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.252@KRBTEST.COM for kudu/127.7.17.252@KRBTEST.COM
I20260927 15:09:55.373963 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.362623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:39549 (local address 127.0.0.1:48056)
0927 15:09:55.362754 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:55.362769 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:55.362863 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:55.364700 (+ 1837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:55.364703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:55.364902 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:55.365006 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:55.365013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.366480 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.366483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:55.367249 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:55.367261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.368381 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.368384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:55.368463 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:55.368931 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:55.368945 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:55.370317 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:55.372303 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:55.372306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:55.372308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:55.372535 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:55.372861 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:55.372864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:55.372866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:55.372963 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:55.373387 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:55.373391 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:55.373503 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 15:09:55.373613 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 15:09:55.373752 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 15:09:55.373998 20780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.363098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48056 (local address 127.7.17.252:39549)
0927 15:09:55.364464 (+ 1366us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:55.364473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:55.364485 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:09:55.364516 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:55.364519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:55.364562 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:55.365117 (+ 555us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:55.365208 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.366372 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.367392 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.368255 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.368661 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:55.370468 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:55.370472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:55.370474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:55.370497 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:55.372181 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.372649 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.372652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.372653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.372732 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.373077 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.373080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.373082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.373282 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:55.373376 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:55.373694 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 15:09:55.373813 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1297,"thread_start_us":327,"threads_started":1}
I20260927 15:09:55.374601 20698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53ebbb8495c54140b19f134bd1cb05c2: Neither blocks nor log segments found. Creating new log.
Sep 27 15:09:55 dist-test-slave-hhpq krb5kdc[20542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.253@KRBTEST.COM
I20260927 15:09:55.375552 20698 log.cc:824] T 00000000000000000000000000000000 P 53ebbb8495c54140b19f134bd1cb05c2: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:55.377107 20779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.367777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:39549 (local address 127.0.0.1:48060)
0927 15:09:55.367872 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:55.367885 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:55.367954 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:55.368376 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:55.368379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:55.368530 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:55.368614 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:55.368621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.369673 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.369675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:55.370452 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:55.370462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.370970 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.370974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:55.371061 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:55.371428 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:55.371440 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:55.372798 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:55.375624 (+ 2826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:55.375626 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:55.375628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:55.375843 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:55.376097 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:55.376100 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:55.376102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:55.376180 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:55.376597 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:55.376600 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:55.376688 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 15:09:55.376782 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 15:09:55.376943 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 15:09:55.377285 20623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb: Bootstrap starting.
I20260927 15:09:55.378008 20781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.367911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48060 (local address 127.7.17.252:39549)
0927 15:09:55.368166 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:55.368173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:55.368185 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:09:55.368214 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:55.368216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:55.368254 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:55.368699 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:55.368803 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.369565 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.370564 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.370867 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.371248 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:55.372953 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:55.372957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:55.372959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:55.372980 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:55.375515 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.375939 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.375941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.375943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.376010 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.376268 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.376271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.376273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.376493 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:55.377644 (+ 1151us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:55.377719 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 15:09:55.377841 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":112,"threads_started":1}
I20260927 15:09:55.378309 20698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53ebbb8495c54140b19f134bd1cb05c2: No bootstrap required, opened a new log
I20260927 15:09:55.379182 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.368305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34598 (local address 127.7.17.253:38055)
0927 15:09:55.368405 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:55.368409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:55.368558 (+ 149us) server_negotiation.cc:407] Connection header received
0927 15:09:55.369034 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:55.369037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:55.369080 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:55.369125 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:55.369901 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.370606 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.372261 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.372598 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.372670 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:55.375812 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:55.375816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:55.375819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:55.375857 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:55.377472 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.377950 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.377954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.377956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.378038 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.378365 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.378368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.378370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.378543 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:55.378706 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:55.378932 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 15:09:55.379045 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:09:55.380685 20623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:55.381271 20775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f2dde15ca784aa6a31b7fbff2cb759e: Bootstrap starting.
I20260927 15:09:55.381778 20623 log.cc:824] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:55.383132 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:09:55.383160 20698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53ebbb8495c54140b19f134bd1cb05c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41865 } } peers { permanent_uuid: "53ebbb8495c54140b19f134bd1cb05c2" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } } peers { permanent_uuid: "9f2dde15ca784aa6a31b7fbff2cb759e" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } }
I20260927 15:09:55.383383 20698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53ebbb8495c54140b19f134bd1cb05c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:55.383430 20698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53ebbb8495c54140b19f134bd1cb05c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53ebbb8495c54140b19f134bd1cb05c2, State: Initialized, Role: FOLLOWER
I20260927 15:09:55.383742 20775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f2dde15ca784aa6a31b7fbff2cb759e: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:55.383896 20698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53ebbb8495c54140b19f134bd1cb05c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41865 } } peers { permanent_uuid: "53ebbb8495c54140b19f134bd1cb05c2" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } } peers { permanent_uuid: "9f2dde15ca784aa6a31b7fbff2cb759e" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } }
I20260927 15:09:55.384416 20775 log.cc:824] T 00000000000000000000000000000000 P 9f2dde15ca784aa6a31b7fbff2cb759e: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:55.384424 20784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ebbb8495c54140b19f134bd1cb05c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41865 } } peers { permanent_uuid: "53ebbb8495c54140b19f134bd1cb05c2" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } } peers { permanent_uuid: "9f2dde15ca784aa6a31b7fbff2cb759e" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } } }
I20260927 15:09:55.384544 20784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ebbb8495c54140b19f134bd1cb05c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:09:55.385378 20698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53ebbb8495c54140b19f134bd1cb05c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:09:55.386555 20775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f2dde15ca784aa6a31b7fbff2cb759e: No bootstrap required, opened a new log
I20260927 15:09:55.392277 20775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f2dde15ca784aa6a31b7fbff2cb759e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41865 } } peers { permanent_uuid: "53ebbb8495c54140b19f134bd1cb05c2" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } } peers { permanent_uuid: "9f2dde15ca784aa6a31b7fbff2cb759e" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } }
I20260927 15:09:55.392675 20775 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f2dde15ca784aa6a31b7fbff2cb759e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:55.392774 20775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f2dde15ca784aa6a31b7fbff2cb759e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f2dde15ca784aa6a31b7fbff2cb759e, State: Initialized, Role: FOLLOWER
I20260927 15:09:55.393432 20775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f2dde15ca784aa6a31b7fbff2cb759e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41865 } } peers { permanent_uuid: "53ebbb8495c54140b19f134bd1cb05c2" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } } peers { permanent_uuid: "9f2dde15ca784aa6a31b7fbff2cb759e" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } }
I20260927 15:09:55.389976 20630 master_runner.cc:186] Error getting master registration for 127.7.17.254:41865: OK, instance_id { permanent_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" instance_seqno: 1790521794895689 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:09:55.394430 20630 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:09:55.394987 20775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f2dde15ca784aa6a31b7fbff2cb759e [sys.catalog]: configured and running, proceeding with master startup.
W20260927 15:09:55.395392 20803 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53ebbb8495c54140b19f134bd1cb05c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 15:09:55.396520 20781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.386233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48064 (local address 127.7.17.252:39549)
0927 15:09:55.386647 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:55.386651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:55.387536 (+ 885us) server_negotiation.cc:407] Connection header received
0927 15:09:55.387713 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:55.387739 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:55.387796 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:55.387860 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:55.388882 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.389622 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.390696 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.391047 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.391117 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:55.392900 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:55.392904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:55.392906 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:55.392936 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:55.394548 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.395063 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.395067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.395069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.395161 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.395459 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.395462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.395464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.395702 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:55.395864 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:55.396241 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 15:09:55.396366 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332}
W20260927 15:09:55.397353 20803 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:09:55.399947 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.386949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34608 (local address 127.7.17.253:38055)
0927 15:09:55.387059 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:55.387064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:55.387957 (+ 893us) server_negotiation.cc:407] Connection header received
0927 15:09:55.388162 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:55.388165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:55.388218 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:55.388271 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:55.389283 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.389974 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.390955 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.391243 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.391309 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:55.396493 (+ 5184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:55.396497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:55.396499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:55.396526 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:55.398140 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.398629 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.398633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.398635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.398724 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.399019 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.399022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.399024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.399239 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:55.399355 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:55.399669 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 15:09:55.399787 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 15:09:55.400739 20816 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f2dde15ca784aa6a31b7fbff2cb759e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:09:55.400843 20816 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:09:55.401229 20805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f2dde15ca784aa6a31b7fbff2cb759e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41865 } } peers { permanent_uuid: "53ebbb8495c54140b19f134bd1cb05c2" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } } peers { permanent_uuid: "9f2dde15ca784aa6a31b7fbff2cb759e" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } } }
I20260927 15:09:55.401444 20805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f2dde15ca784aa6a31b7fbff2cb759e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:09:55.399240 20705 master_runner.cc:186] Error getting master registration for 127.7.17.254:41865: OK, instance_id { permanent_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" instance_seqno: 1790521794895689 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:09:55.402931 20705 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:09:55.405390 20627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:55.385487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60602 (local address 127.7.17.254:41865)
0927 15:09:55.385605 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:55.385609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:55.386170 (+ 561us) server_negotiation.cc:407] Connection header received
0927 15:09:55.386370 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:55.386373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:55.386428 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:55.386482 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:55.387444 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.388227 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.395218 (+ 6991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:55.395545 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:55.395631 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:55.398868 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:55.398872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:55.398874 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:55.398902 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:55.403549 (+ 4647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.403990 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.403994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.403996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.404103 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:55.404426 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:55.404430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:55.404432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:55.404673 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:55.404806 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:55.405113 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 15:09:55.405239 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:09:55.406442 20623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb: No bootstrap required, opened a new log
I20260927 15:09:55.410094 20623 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41865 } } peers { permanent_uuid: "53ebbb8495c54140b19f134bd1cb05c2" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } } peers { permanent_uuid: "9f2dde15ca784aa6a31b7fbff2cb759e" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } }
I20260927 15:09:55.410317 20623 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:55.410398 20623 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ff10b7dcdb4079b21c4ad9c8bc53fb, State: Initialized, Role: FOLLOWER
I20260927 15:09:55.410840 20623 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41865 } } peers { permanent_uuid: "53ebbb8495c54140b19f134bd1cb05c2" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } } peers { permanent_uuid: "9f2dde15ca784aa6a31b7fbff2cb759e" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } }
I20260927 15:09:55.411346 20818 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41865 } } peers { permanent_uuid: "53ebbb8495c54140b19f134bd1cb05c2" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } } peers { permanent_uuid: "9f2dde15ca784aa6a31b7fbff2cb759e" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } } }
I20260927 15:09:55.411527 20818 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:09:55.412213 20623 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [sys.catalog]: configured and running, proceeding with master startup.
W20260927 15:09:55.420231 20829 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:09:55.420421 20829 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:09:55.421912 20555 master_runner.cc:263] No leader master found from master c2ff10b7dcdb4079b21c4ad9c8bc53fb
I20260927 15:09:55.421998 20555 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:09:55.456470 20818 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:09:55.456683 20818 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41865 } } peers { permanent_uuid: "53ebbb8495c54140b19f134bd1cb05c2" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } } peers { permanent_uuid: "9f2dde15ca784aa6a31b7fbff2cb759e" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } }
I20260927 15:09:55.457873 20818 leader_election.cc:288] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53ebbb8495c54140b19f134bd1cb05c2 (127.7.17.253:38055), 9f2dde15ca784aa6a31b7fbff2cb759e (127.7.17.252:39549)
I20260927 15:09:55.458679 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f2dde15ca784aa6a31b7fbff2cb759e" is_pre_election: true
I20260927 15:09:55.459059 20749 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9f2dde15ca784aa6a31b7fbff2cb759e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ff10b7dcdb4079b21c4ad9c8bc53fb in term 0.
I20260927 15:09:55.459667 20572 leader_election.cc:302] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f2dde15ca784aa6a31b7fbff2cb759e, c2ff10b7dcdb4079b21c4ad9c8bc53fb; no voters:
I20260927 15:09:55.460763 20672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53ebbb8495c54140b19f134bd1cb05c2" is_pre_election: true
I20260927 15:09:55.461105 20672 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 53ebbb8495c54140b19f134bd1cb05c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ff10b7dcdb4079b21c4ad9c8bc53fb in term 0.
I20260927 15:09:55.463357 20818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:09:55.463464 20818 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:09:55.463505 20818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:55.465564 20818 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41865 } } peers { permanent_uuid: "53ebbb8495c54140b19f134bd1cb05c2" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } } peers { permanent_uuid: "9f2dde15ca784aa6a31b7fbff2cb759e" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } }
I20260927 15:09:55.466040 20818 leader_election.cc:288] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [CANDIDATE]: Term 1 election: Requested vote from peers 53ebbb8495c54140b19f134bd1cb05c2 (127.7.17.253:38055), 9f2dde15ca784aa6a31b7fbff2cb759e (127.7.17.252:39549)
I20260927 15:09:55.466660 20749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f2dde15ca784aa6a31b7fbff2cb759e"
I20260927 15:09:55.466804 20749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f2dde15ca784aa6a31b7fbff2cb759e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:55.468210 20749 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9f2dde15ca784aa6a31b7fbff2cb759e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ff10b7dcdb4079b21c4ad9c8bc53fb in term 1.
I20260927 15:09:55.468793 20672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53ebbb8495c54140b19f134bd1cb05c2"
I20260927 15:09:55.468952 20672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53ebbb8495c54140b19f134bd1cb05c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:55.470235 20672 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 53ebbb8495c54140b19f134bd1cb05c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ff10b7dcdb4079b21c4ad9c8bc53fb in term 1.
I20260927 15:09:55.468611 20572 leader_election.cc:302] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f2dde15ca784aa6a31b7fbff2cb759e, c2ff10b7dcdb4079b21c4ad9c8bc53fb; no voters:
I20260927 15:09:55.471040 20818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:09:55.471326 20818 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [term 1 LEADER]: Becoming Leader. State: Replica: c2ff10b7dcdb4079b21c4ad9c8bc53fb, State: Running, Role: LEADER
I20260927 15:09:55.471772 20818 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41865 } } peers { permanent_uuid: "53ebbb8495c54140b19f134bd1cb05c2" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } } peers { permanent_uuid: "9f2dde15ca784aa6a31b7fbff2cb759e" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } }
I20260927 15:09:55.478581 20818 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2ff10b7dcdb4079b21c4ad9c8bc53fb. Latest consensus state: current_term: 1 leader_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41865 } } peers { permanent_uuid: "53ebbb8495c54140b19f134bd1cb05c2" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } } peers { permanent_uuid: "9f2dde15ca784aa6a31b7fbff2cb759e" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } } }
I20260927 15:09:55.478849 20818 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [sys.catalog]: This master's current role is: LEADER
I20260927 15:09:55.479573 20833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:09:55.480651 20833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:09:55.489665 20749 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9f2dde15ca784aa6a31b7fbff2cb759e [term 1 FOLLOWER]: Refusing update from remote peer c2ff10b7dcdb4079b21c4ad9c8bc53fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:09:55.490677 20831 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f2dde15ca784aa6a31b7fbff2cb759e" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:09:55.498657 20805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f2dde15ca784aa6a31b7fbff2cb759e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2ff10b7dcdb4079b21c4ad9c8bc53fb. Latest consensus state: current_term: 1 leader_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41865 } } peers { permanent_uuid: "53ebbb8495c54140b19f134bd1cb05c2" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } } peers { permanent_uuid: "9f2dde15ca784aa6a31b7fbff2cb759e" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } } }
I20260927 15:09:55.498831 20805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f2dde15ca784aa6a31b7fbff2cb759e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:09:55.500478 20672 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53ebbb8495c54140b19f134bd1cb05c2 [term 1 FOLLOWER]: Refusing update from remote peer c2ff10b7dcdb4079b21c4ad9c8bc53fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:09:55.501325 20831 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "53ebbb8495c54140b19f134bd1cb05c2" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:09:55.506600 20784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ebbb8495c54140b19f134bd1cb05c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2ff10b7dcdb4079b21c4ad9c8bc53fb. Latest consensus state: current_term: 1 leader_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41865 } } peers { permanent_uuid: "53ebbb8495c54140b19f134bd1cb05c2" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } } peers { permanent_uuid: "9f2dde15ca784aa6a31b7fbff2cb759e" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } } }
I20260927 15:09:55.506778 20784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ebbb8495c54140b19f134bd1cb05c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:09:55.515229 20831 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41865 } } peers { permanent_uuid: "53ebbb8495c54140b19f134bd1cb05c2" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } } peers { permanent_uuid: "9f2dde15ca784aa6a31b7fbff2cb759e" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } } }
I20260927 15:09:55.515417 20831 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [sys.catalog]: This master's current role is: LEADER
I20260927 15:09:55.517059 20784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ebbb8495c54140b19f134bd1cb05c2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41865 } } peers { permanent_uuid: "53ebbb8495c54140b19f134bd1cb05c2" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } } peers { permanent_uuid: "9f2dde15ca784aa6a31b7fbff2cb759e" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } } }
I20260927 15:09:55.517204 20784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ebbb8495c54140b19f134bd1cb05c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:09:55.517764 20834 mvcc.cc:205] Tried to move back new op lower bound from 7333977274315345920 to 7333977274256764928. Current Snapshot: MvccSnapshot[applied={T|T < 7333977274315345920}]
I20260927 15:09:55.519851 20837 mvcc.cc:205] Tried to move back new op lower bound from 7333977274315345920 to 7333977274256764928. Current Snapshot: MvccSnapshot[applied={T|T < 7333977274315345920}]
I20260927 15:09:55.521775 20818 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41865 } } peers { permanent_uuid: "53ebbb8495c54140b19f134bd1cb05c2" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } } peers { permanent_uuid: "9f2dde15ca784aa6a31b7fbff2cb759e" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } } }
I20260927 15:09:55.521938 20818 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb [sys.catalog]: This master's current role is: LEADER
I20260927 15:09:55.523322 20805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f2dde15ca784aa6a31b7fbff2cb759e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff10b7dcdb4079b21c4ad9c8bc53fb" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 41865 } } peers { permanent_uuid: "53ebbb8495c54140b19f134bd1cb05c2" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38055 } } peers { permanent_uuid: "9f2dde15ca784aa6a31b7fbff2cb759e" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 39549 } } }
I20260927 15:09:55.523474 20805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f2dde15ca784aa6a31b7fbff2cb759e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:09:55.531960 20835 mvcc.cc:205] Tried to move back new op lower bound from 7333977274315345920 to 7333977274256764928. Current Snapshot: MvccSnapshot[applied={T|T < 7333977274315345920}]
I20260927 15:09:55.533284 20833 catalog_manager.cc:1347] Generated new cluster ID: ac93edf3a74d4decb6783499ef79ce06
I20260927 15:09:55.533375 20833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:09:55.592306 20833 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:09:55.593457 20833 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:09:55.605330 20833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2ff10b7dcdb4079b21c4ad9c8bc53fb: Generated new TSK 0
I20260927 15:09:55.607128 20833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:09:56.400293 20803 catalog_manager.cc:1259] Loaded cluster ID: ac93edf3a74d4decb6783499ef79ce06
I20260927 15:09:56.400386 20803 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 53ebbb8495c54140b19f134bd1cb05c2: loading cluster ID for follower catalog manager: success
I20260927 15:09:56.402177 20816 catalog_manager.cc:1259] Loaded cluster ID: ac93edf3a74d4decb6783499ef79ce06
I20260927 15:09:56.402171 20803 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 53ebbb8495c54140b19f134bd1cb05c2: acquiring CA information for follower catalog manager: success
I20260927 15:09:56.402252 20816 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9f2dde15ca784aa6a31b7fbff2cb759e: loading cluster ID for follower catalog manager: success
I20260927 15:09:56.403453 20803 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 53ebbb8495c54140b19f134bd1cb05c2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:09:56.403920 20816 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9f2dde15ca784aa6a31b7fbff2cb759e: acquiring CA information for follower catalog manager: success
I20260927 15:09:56.405103 20816 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9f2dde15ca784aa6a31b7fbff2cb759e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:09:56.677183 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 20555
I20260927 15:09:56.682637 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 20630
I20260927 15:09:56.687762 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 20705
2026-09-27T15:09:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3053 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:09:56 dist-test-slave-hhpq krb5kdc[20847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:09:56 dist-test-slave-hhpq krb5kdc[20847](info): set up 2 sockets
Sep 27 15:09:56 dist-test-slave-hhpq krb5kdc[20847](info): commencing operation
krb5kdc: starting...
W20260927 15:09:57.644919 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:09:57 dist-test-slave-hhpq krb5kdc[20847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:09:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:09:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:09:57.768177 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:38605
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:41311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:38605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:09:57.880081 20860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:57.880388 20860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:57.880430 20860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:57.884641 20860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:09:57.884721 20860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:57.884740 20860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:57.884756 20860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:09:57.884770 20860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:09:57.890407 20860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:41311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:38605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:38605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:09:57.891534 20860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:57.892550 20860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:57.901495 20868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:57.901471 20865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:57.901698 20860 server_base.cc:1034] running on GCE node
W20260927 15:09:57.901471 20866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:57.902669 20860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:57.903993 20860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:57.905206 20860 hybrid_clock.cc:648] HybridClock initialized: now 1790521797905181 us; error 44 us; skew 500 ppm
Sep 27 15:09:57 dist-test-slave-hhpq krb5kdc[20847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521797, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:57.908226 20860 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:09:57.909323 20860 webserver.cc:460] Webserver started at http://127.7.17.254:46385/ using document root <none> and password file <none>
I20260927 15:09:57.909929 20860 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:57.909996 20860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:57.910202 20860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:09:57.912426 20860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "8f9a456f106d458594fe7ee6104bd9d5"
format_stamp: "Formatted at 2026-09-27 15:09:57 on dist-test-slave-hhpq"
server_key: "c826239b2182f232d8f8423560f51680"
server_key_iv: "6c48ce7ed139b9658cc3a732e7a06344"
server_key_version: "encryptionkey@0"
I20260927 15:09:57.913038 20860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "8f9a456f106d458594fe7ee6104bd9d5"
format_stamp: "Formatted at 2026-09-27 15:09:57 on dist-test-slave-hhpq"
server_key: "c826239b2182f232d8f8423560f51680"
server_key_iv: "6c48ce7ed139b9658cc3a732e7a06344"
server_key_version: "encryptionkey@0"
I20260927 15:09:57.917146 20860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:09:57.919828 20875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:57.921339 20860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:09:57.921473 20860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "8f9a456f106d458594fe7ee6104bd9d5"
format_stamp: "Formatted at 2026-09-27 15:09:57 on dist-test-slave-hhpq"
server_key: "c826239b2182f232d8f8423560f51680"
server_key_iv: "6c48ce7ed139b9658cc3a732e7a06344"
server_key_version: "encryptionkey@0"
I20260927 15:09:57.921595 20860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:57.935279 20860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:57.938612 20860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:57.938807 20860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:57.945808 20860 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:38605
I20260927 15:09:57.945892 20927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:38605 every 8 connection(s)
I20260927 15:09:57.946842 20860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:09:57.949992 20928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:57.954691 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 20860
I20260927 15:09:57.954825 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:09:57.955065 7239 external_mini_cluster.cc:1428] Setting key e20c09b10ba8d818f2d2681f4adf3caa
I20260927 15:09:57.956203 20928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5: Bootstrap starting.
I20260927 15:09:57.958616 20928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:57.959539 20928 log.cc:824] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:57.961814 20928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5: No bootstrap required, opened a new log
Sep 27 15:09:57 dist-test-slave-hhpq krb5kdc[20847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:09:57.966176 20928 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9a456f106d458594fe7ee6104bd9d5" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38605 } }
I20260927 15:09:57.966451 20928 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:57.966539 20928 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9a456f106d458594fe7ee6104bd9d5, State: Initialized, Role: FOLLOWER
I20260927 15:09:57.967024 20928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9a456f106d458594fe7ee6104bd9d5" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38605 } }
I20260927 15:09:57.967162 20928 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:09:57.967226 20928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:09:57.967325 20928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:57.968485 20928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9a456f106d458594fe7ee6104bd9d5" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38605 } }
I20260927 15:09:57.968859 20928 leader_election.cc:302] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f9a456f106d458594fe7ee6104bd9d5; no voters:
I20260927 15:09:57.969252 20928 leader_election.cc:288] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:09:57.969380 20933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:09:57.969673 20933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8f9a456f106d458594fe7ee6104bd9d5, State: Running, Role: LEADER
I20260927 15:09:57.970121 20933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9a456f106d458594fe7ee6104bd9d5" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38605 } }
I20260927 15:09:57.970628 20928 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:09:57.970877 20931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:57.956525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52696 (local address 127.7.17.254:38605)
0927 15:09:57.956976 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:57.956993 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:57.957033 (+ 40us) server_negotiation.cc:407] Connection header received
0927 15:09:57.957720 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:57.957740 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:57.958165 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:57.958468 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:57.959429 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:57.960533 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:57.961738 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:57.962312 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:57.962457 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:57.965507 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:57.965513 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:57.965531 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:57.965567 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:57.967354 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:57.967897 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:57.967902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:57.967907 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:57.968037 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:57.968327 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:57.968329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:57.968330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:57.968792 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:57.969023 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:57.969379 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 15:09:57.969738 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":118,"threads_started":1}
I20260927 15:09:57.972931 20934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f9a456f106d458594fe7ee6104bd9d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9a456f106d458594fe7ee6104bd9d5" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38605 } } }
I20260927 15:09:57.973063 20934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:09:57.973588 20935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f9a456f106d458594fe7ee6104bd9d5. Latest consensus state: current_term: 1 leader_uuid: "8f9a456f106d458594fe7ee6104bd9d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9a456f106d458594fe7ee6104bd9d5" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38605 } } }
I20260927 15:09:57.973686 20935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:09:57.974220 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 15:09:57.975967 20947 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:09:57.976082 20947 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:09:57.976146 20948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:09:57.977041 20948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:09:57.983413 20948 catalog_manager.cc:1347] Generated new cluster ID: db64925bd32141389cd61833fd8619bb
I20260927 15:09:57.983506 20948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:09:57.997385 20948 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:09:57.998461 20948 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:09:58.004971 20948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f9a456f106d458594fe7ee6104bd9d5: Generated new TSK 0
I20260927 15:09:58.005736 20948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:09:58.030735 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 20860
2026-09-27T15:09:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1341 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:09:58 dist-test-slave-hhpq krb5kdc[20955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:09:58 dist-test-slave-hhpq krb5kdc[20955](info): set up 2 sockets
Sep 27 15:09:58 dist-test-slave-hhpq krb5kdc[20955](info): commencing operation
krb5kdc: starting...
W20260927 15:09:58.989027 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:09:59 dist-test-slave-hhpq krb5kdc[20955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:09:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:09:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:09:59.112018 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:44411
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:44411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:09:59.235110 20968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:59.235410 20968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:59.235451 20968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:59.239571 20968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:09:59.239641 20968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:59.239660 20968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:59.239677 20968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:09:59.239691 20968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:09:59.245179 20968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:44411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:44411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:09:59.246275 20968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:59.247280 20968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:59.256268 20973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:59.256275 20976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:59.256348 20974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:59.256338 20968 server_base.cc:1034] running on GCE node
I20260927 15:09:59.257174 20968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:59.258352 20968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:59.259557 20968 hybrid_clock.cc:648] HybridClock initialized: now 1790521799259540 us; error 39 us; skew 500 ppm
Sep 27 15:09:59 dist-test-slave-hhpq krb5kdc[20955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521799, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:59.262692 20968 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:09:59.263710 20968 webserver.cc:460] Webserver started at http://127.7.17.254:43913/ using document root <none> and password file <none>
I20260927 15:09:59.264285 20968 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:59.264357 20968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:59.264568 20968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:09:59.266727 20968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "f13f064244294db4b21bd6bf73cf2da4"
format_stamp: "Formatted at 2026-09-27 15:09:59 on dist-test-slave-hhpq"
server_key: "a92b9600e0e9d7a7d7b1c1183e037136"
server_key_iv: "c4743d41bcfb1755983e65f0d82596fb"
server_key_version: "encryptionkey@0"
I20260927 15:09:59.267302 20968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "f13f064244294db4b21bd6bf73cf2da4"
format_stamp: "Formatted at 2026-09-27 15:09:59 on dist-test-slave-hhpq"
server_key: "a92b9600e0e9d7a7d7b1c1183e037136"
server_key_iv: "c4743d41bcfb1755983e65f0d82596fb"
server_key_version: "encryptionkey@0"
I20260927 15:09:59.271333 20968 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:09:59.274016 20983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:59.275384 20968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:09:59.275527 20968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "f13f064244294db4b21bd6bf73cf2da4"
format_stamp: "Formatted at 2026-09-27 15:09:59 on dist-test-slave-hhpq"
server_key: "a92b9600e0e9d7a7d7b1c1183e037136"
server_key_iv: "c4743d41bcfb1755983e65f0d82596fb"
server_key_version: "encryptionkey@0"
I20260927 15:09:59.275642 20968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:59.304760 20968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:59.308348 20968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:59.308547 20968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:59.315871 20968 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:44411
I20260927 15:09:59.316159 21035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:44411 every 8 connection(s)
I20260927 15:09:59.316969 20968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:09:59.319058 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 20968
I20260927 15:09:59.319221 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:09:59.319514 7239 external_mini_cluster.cc:1428] Setting key 8301bc2acac3fd8dfd9beb3214295b1c
I20260927 15:09:59.320358 21036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:09:59.327363 21036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4: Bootstrap starting.
Sep 27 15:09:59 dist-test-slave-hhpq krb5kdc[20955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:09:59.330344 21036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4: Neither blocks nor log segments found. Creating new log.
I20260927 15:09:59.331349 21036 log.cc:824] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4: Log is configured to *not* fsync() on all Append() calls
I20260927 15:09:59.334439 21036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4: No bootstrap required, opened a new log
I20260927 15:09:59.335525 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:59.321021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33092 (local address 127.7.17.254:44411)
0927 15:09:59.321446 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:59.321463 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:59.321501 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:09:59.322167 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:59.322185 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:59.322542 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:59.322860 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:59.323833 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.324896 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.326153 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.326667 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.326864 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:59.329896 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:59.329904 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:59.329922 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:59.329958 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:59.332168 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:59.332692 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:59.332698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:59.332704 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:59.332902 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:59.333225 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:59.333229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:59.333240 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:59.333693 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:59.333957 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:59.334329 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 15:09:59.334674 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":129,"threads_started":1}
I20260927 15:09:59.338160 21036 raft_consensus.cc:348] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13f064244294db4b21bd6bf73cf2da4" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 44411 } }
I20260927 15:09:59.338452 21036 raft_consensus.cc:374] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:09:59.338526 21036 raft_consensus.cc:729] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f13f064244294db4b21bd6bf73cf2da4, State: Initialized, Role: FOLLOWER
I20260927 15:09:59.338950 21036 consensus_queue.cc:260] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13f064244294db4b21bd6bf73cf2da4" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 44411 } }
I20260927 15:09:59.339076 21036 raft_consensus.cc:388] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:09:59.339121 21036 raft_consensus.cc:482] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:09:59.339193 21036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:09:59.340155 21036 raft_consensus.cc:504] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13f064244294db4b21bd6bf73cf2da4" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 44411 } }
I20260927 15:09:59.340472 21036 leader_election.cc:302] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f13f064244294db4b21bd6bf73cf2da4; no voters:
I20260927 15:09:59.340768 21036 leader_election.cc:288] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:09:59.340896 21041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:09:59.341140 21041 raft_consensus.cc:686] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 1 LEADER]: Becoming Leader. State: Replica: f13f064244294db4b21bd6bf73cf2da4, State: Running, Role: LEADER
I20260927 15:09:59.341609 21041 consensus_queue.cc:237] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13f064244294db4b21bd6bf73cf2da4" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 44411 } }
I20260927 15:09:59.342145 21036 sys_catalog.cc:564] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:09:59.343238 21043 sys_catalog.cc:455] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f13f064244294db4b21bd6bf73cf2da4. Latest consensus state: current_term: 1 leader_uuid: "f13f064244294db4b21bd6bf73cf2da4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13f064244294db4b21bd6bf73cf2da4" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 44411 } } }
I20260927 15:09:59.343385 21043 sys_catalog.cc:458] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [sys.catalog]: This master's current role is: LEADER
I20260927 15:09:59.343724 21042 sys_catalog.cc:455] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f13f064244294db4b21bd6bf73cf2da4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13f064244294db4b21bd6bf73cf2da4" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 44411 } } }
I20260927 15:09:59.343802 21050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:09:59.343812 21042 sys_catalog.cc:458] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [sys.catalog]: This master's current role is: LEADER
I20260927 15:09:59.346822 21050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:09:59.352560 21050 catalog_manager.cc:1347] Generated new cluster ID: 313a2279e2cd4e638c59a691842e4f68
I20260927 15:09:59.352659 21050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:09:59.359265 21050 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:09:59.360711 21050 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:09:59.367507 21050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4: Generated new TSK 0
I20260927 15:09:59.368251 21050 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
I20260927 15:09:59.408425 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:44411
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:09:59.525393 21062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:59.525694 21062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:59.525760 21062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:59.530045 21062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:59.530134 21062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:59.530220 21062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:09:59.535899 21062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:44411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:09:59.537114 21062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:59.538167 21062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:09:59.548262 21062 server_base.cc:1034] running on GCE node
W20260927 15:09:59.548239 21068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:59.548257 21067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:59.548307 21070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:59.548998 21062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:59.549806 21062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:59.551002 21062 hybrid_clock.cc:648] HybridClock initialized: now 1790521799550976 us; error 35 us; skew 500 ppm
Sep 27 15:09:59 dist-test-slave-hhpq krb5kdc[20955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521799, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:59.554073 21062 init.cc:377] Logged in from keytab as kudu/127.7.17.193@KRBTEST.COM (short username kudu)
I20260927 15:09:59.555150 21062 webserver.cc:460] Webserver started at http://127.7.17.193:34993/ using document root <none> and password file <none>
I20260927 15:09:59.555704 21062 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:59.555764 21062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:59.555948 21062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:09:59.558241 21062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "881cc60f613d4b408db7a4c7189ef7ff"
format_stamp: "Formatted at 2026-09-27 15:09:59 on dist-test-slave-hhpq"
server_key: "53a2aa6ca8989f8136f80a8d47d45b2d"
server_key_iv: "b024aea45856006327588994891f1540"
server_key_version: "encryptionkey@0"
I20260927 15:09:59.558851 21062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "881cc60f613d4b408db7a4c7189ef7ff"
format_stamp: "Formatted at 2026-09-27 15:09:59 on dist-test-slave-hhpq"
server_key: "53a2aa6ca8989f8136f80a8d47d45b2d"
server_key_iv: "b024aea45856006327588994891f1540"
server_key_version: "encryptionkey@0"
I20260927 15:09:59.563233 21062 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:09:59.566267 21077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:59.567742 21062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:09:59.567931 21062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "881cc60f613d4b408db7a4c7189ef7ff"
format_stamp: "Formatted at 2026-09-27 15:09:59 on dist-test-slave-hhpq"
server_key: "53a2aa6ca8989f8136f80a8d47d45b2d"
server_key_iv: "b024aea45856006327588994891f1540"
server_key_version: "encryptionkey@0"
I20260927 15:09:59.568045 21062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:59.589076 21062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:59.592626 21062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:59.592818 21062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:59.593493 21062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:09:59.594388 21062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:09:59.594444 21062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:59.594486 21062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:09:59.594509 21062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:59.603858 21062 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:43849
I20260927 15:09:59.603938 21190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:43849 every 8 connection(s)
I20260927 15:09:59.604882 21062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:09:59.605895 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 21062
I20260927 15:09:59.605997 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:09:59.606462 7239 external_mini_cluster.cc:1428] Setting key 7988804682b2b5ab1cd220a76dfe7107
WARNING: no policy specified for kudu/127.7.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.194@KRBTEST.COM" created.
Sep 27 15:09:59 dist-test-slave-hhpq krb5kdc[20955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521799, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:09:59.623497 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:59.606959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:51131 (local address 127.7.17.254:44411)
0927 15:09:59.607121 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:59.607124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:59.608514 (+ 1390us) server_negotiation.cc:407] Connection header received
0927 15:09:59.609790 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:59.609795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:59.609858 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:59.609953 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:59.611257 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.611904 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.613223 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.613474 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.613570 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:59.619434 (+ 5864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:59.619438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:59.619440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:59.619472 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:59.621208 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:59.621820 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:59.621823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:59.621825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:59.621915 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:59.622284 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:59.622287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:59.622289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:59.622500 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:59.622588 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:59.623175 (+ 587us) server_negotiation.cc:299] Negotiation successful
0927 15:09:59.623326 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:09:59.624387 21193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:59.607455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.193:51131)
0927 15:09:59.608327 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:59.608393 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:59.609512 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:59.610159 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:59.610168 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:59.610558 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:59.611103 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:59.611118 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.612038 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.612042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:59.613098 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:59.613109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.613632 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.613636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:59.613841 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:59.614598 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:59.614616 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:59.619177 (+ 4561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:59.621366 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:59.621374 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:59.621393 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:59.621698 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:59.622003 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:59.622006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:59.622008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:59.622179 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:59.622608 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:59.622615 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:59.622918 (+ 303us) client_negotiation.cc:769] Sending connection context
0927 15:09:59.623094 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 15:09:59.623482 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":153,"threads_started":1}
I20260927 15:09:59.625957 21191 heartbeater.cc:347] Connected to a master server at 127.7.17.254:44411
I20260927 15:09:59.626235 21191 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:59.626772 21191 heartbeater.cc:511] Master 127.7.17.254:44411 requested a full tablet report, sending...
I20260927 15:09:59.628666 20996 ts_manager.cc:194] Registered new tserver with Master: 881cc60f613d4b408db7a4c7189ef7ff (127.7.17.193:43849)
I20260927 15:09:59.629895 20996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:51131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
I20260927 15:09:59.637323 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:44411
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:09:59.751152 21196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:59.751454 21196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:59.751503 21196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:59.755683 21196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:59.755760 21196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:59.755831 21196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:09:59.761428 21196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:44411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:09:59.762565 21196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:59.763624 21196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:59.773357 21202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:59.773368 21204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:59.773401 21201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:59.773578 21196 server_base.cc:1034] running on GCE node
I20260927 15:09:59.774070 21196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:59.774740 21196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:59.775928 21196 hybrid_clock.cc:648] HybridClock initialized: now 1790521799775904 us; error 39 us; skew 500 ppm
Sep 27 15:09:59 dist-test-slave-hhpq krb5kdc[20955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521799, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:59.778769 21196 init.cc:377] Logged in from keytab as kudu/127.7.17.194@KRBTEST.COM (short username kudu)
I20260927 15:09:59.779800 21196 webserver.cc:460] Webserver started at http://127.7.17.194:40863/ using document root <none> and password file <none>
I20260927 15:09:59.780440 21196 fs_manager.cc:359] Metadata directory not provided
I20260927 15:09:59.780514 21196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:09:59.780723 21196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:09:59.782862 21196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "8f73c2d3aac64266bf45eee00a10c095"
format_stamp: "Formatted at 2026-09-27 15:09:59 on dist-test-slave-hhpq"
server_key: "961d8a9a99c24315704f0c53399c3f39"
server_key_iv: "2937772d44511fc1862e8fc36057298a"
server_key_version: "encryptionkey@0"
I20260927 15:09:59.783452 21196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "8f73c2d3aac64266bf45eee00a10c095"
format_stamp: "Formatted at 2026-09-27 15:09:59 on dist-test-slave-hhpq"
server_key: "961d8a9a99c24315704f0c53399c3f39"
server_key_iv: "2937772d44511fc1862e8fc36057298a"
server_key_version: "encryptionkey@0"
I20260927 15:09:59.787380 21196 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:09:59.790124 21211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:59.791569 21196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:09:59.791704 21196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "8f73c2d3aac64266bf45eee00a10c095"
format_stamp: "Formatted at 2026-09-27 15:09:59 on dist-test-slave-hhpq"
server_key: "961d8a9a99c24315704f0c53399c3f39"
server_key_iv: "2937772d44511fc1862e8fc36057298a"
server_key_version: "encryptionkey@0"
I20260927 15:09:59.791828 21196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:09:59.803869 21196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:09:59.807169 21196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:09:59.807376 21196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:09:59.808022 21196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:09:59.808954 21196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:09:59.809007 21196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:59.809062 21196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:09:59.809088 21196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:09:59.818019 21196 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:42553
I20260927 15:09:59.818042 21324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:42553 every 8 connection(s)
I20260927 15:09:59.818969 21196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:09:59.823855 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 21196
I20260927 15:09:59.824003 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:09:59.824313 7239 external_mini_cluster.cc:1428] Setting key bc37a0b0b3e8693f5a65267913b61513
Sep 27 15:09:59 dist-test-slave-hhpq krb5kdc[20955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521799, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.17.195@KRBTEST.COM; defaulting to no policy
I20260927 15:09:59.835608 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:59.820942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:38863 (local address 127.7.17.254:44411)
0927 15:09:59.821079 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 15:09:59.821083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:09:59.822309 (+ 1226us) server_negotiation.cc:407] Connection header received
0927 15:09:59.823251 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:09:59.823255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:09:59.823318 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:09:59.823396 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:09:59.825075 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.825779 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.827465 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.827770 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.827881 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:59.830806 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:09:59.830811 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:09:59.830814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:09:59.830848 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:09:59.832685 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:59.833356 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:59.833360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:59.833370 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:59.833471 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:09:59.833828 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:09:59.833832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:09:59.833834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:09:59.834102 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:09:59.834208 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 15:09:59.835330 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0927 15:09:59.835456 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:09:59.836411 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:09:59.821330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.194:38863)
0927 15:09:59.822116 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0927 15:09:59.822164 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:09:59.823018 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:09:59.823617 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:09:59.823628 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:09:59.824165 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:09:59.824910 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:59.824934 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.825967 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.825971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:59.827328 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:09:59.827341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:09:59.827938 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:09:59.827941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:09:59.828100 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:09:59.828744 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:09:59.828762 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:09:59.830529 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:09:59.832894 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:59.832901 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:59.832917 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:59.833219 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:59.833563 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:09:59.833565 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:09:59.833567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:09:59.833730 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:09:59.834218 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:09:59.834225 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:09:59.834940 (+ 715us) client_negotiation.cc:769] Sending connection context
0927 15:09:59.835158 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 15:09:59.835474 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":129,"threads_started":1}
I20260927 15:09:59.837743 21325 heartbeater.cc:347] Connected to a master server at 127.7.17.254:44411
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.195@KRBTEST.COM" created.
I20260927 15:09:59.838042 21325 heartbeater.cc:464] Registering TS with master...
I20260927 15:09:59.838676 21325 heartbeater.cc:511] Master 127.7.17.254:44411 requested a full tablet report, sending...
I20260927 15:09:59.840077 20996 ts_manager.cc:194] Registered new tserver with Master: 8f73c2d3aac64266bf45eee00a10c095 (127.7.17.194:42553)
I20260927 15:09:59.840983 20996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:38863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
I20260927 15:09:59.856294 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:44411
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:09:59.971061 21330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:09:59.971355 21330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:09:59.971400 21330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:09:59.975740 21330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:09:59.975838 21330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:09:59.975903 21330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:09:59.981693 21330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:44411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:09:59.982868 21330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:09:59.983935 21330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:09:59.994105 21335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:59.994290 21330 server_base.cc:1034] running on GCE node
W20260927 15:09:59.994099 21336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:09:59.994100 21338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:09:59.994800 21330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:09:59.995548 21330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:09:59.996740 21330 hybrid_clock.cc:648] HybridClock initialized: now 1790521799996737 us; error 61 us; skew 500 ppm
Sep 27 15:09:59 dist-test-slave-hhpq krb5kdc[20955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521799, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:09:59.999598 21330 init.cc:377] Logged in from keytab as kudu/127.7.17.195@KRBTEST.COM (short username kudu)
I20260927 15:10:00.000905 21330 webserver.cc:460] Webserver started at http://127.7.17.195:46373/ using document root <none> and password file <none>
I20260927 15:10:00.001513 21330 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:00.001577 21330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:00.001778 21330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:10:00.004223 21330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "8fac9d4b8f204372b80178dac5aa38cd"
format_stamp: "Formatted at 2026-09-27 15:09:59 on dist-test-slave-hhpq"
server_key: "bf99e66287c79017e81df9bc48ab2d7c"
server_key_iv: "9201de33ac3b0045d787800f99775c6f"
server_key_version: "encryptionkey@0"
I20260927 15:10:00.004845 21330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "8fac9d4b8f204372b80178dac5aa38cd"
format_stamp: "Formatted at 2026-09-27 15:09:59 on dist-test-slave-hhpq"
server_key: "bf99e66287c79017e81df9bc48ab2d7c"
server_key_iv: "9201de33ac3b0045d787800f99775c6f"
server_key_version: "encryptionkey@0"
I20260927 15:10:00.008785 21330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:10:00.011546 21345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:00.013047 21330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:10:00.013160 21330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "8fac9d4b8f204372b80178dac5aa38cd"
format_stamp: "Formatted at 2026-09-27 15:09:59 on dist-test-slave-hhpq"
server_key: "bf99e66287c79017e81df9bc48ab2d7c"
server_key_iv: "9201de33ac3b0045d787800f99775c6f"
server_key_version: "encryptionkey@0"
I20260927 15:10:00.013262 21330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:00.024729 21330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:00.028297 21330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:00.028473 21330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:00.029117 21330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:00.029984 21330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:10:00.030031 21330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:00.030071 21330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:10:00.030092 21330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:00.038897 21330 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:42987
I20260927 15:10:00.039000 21458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:42987 every 8 connection(s)
I20260927 15:10:00.039847 21330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:10:00.043135 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 21330
I20260927 15:10:00.043270 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:10:00.043653 7239 external_mini_cluster.cc:1428] Setting key 95b3cc48adedba3dc237d39662810756
Sep 27 15:10:00 dist-test-slave-hhpq krb5kdc[20955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521799, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:10:00.054637 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:00.041807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:49481 (local address 127.7.17.254:44411)
0927 15:10:00.041932 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:00.041935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:00.042923 (+ 988us) server_negotiation.cc:407] Connection header received
0927 15:10:00.044378 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:00.044381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:00.044441 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:00.044526 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:00.045970 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.046516 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.047767 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.048000 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.048079 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:00.050787 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:00.050791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:00.050793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:00.050823 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:00.052262 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:00.052878 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:00.052882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:00.052884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:00.052988 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:00.053409 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:00.053413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:00.053414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:00.053678 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:00.053769 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:00.054380 (+ 611us) server_negotiation.cc:299] Negotiation successful
0927 15:10:00.054511 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:10:00.055533 21461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:00.042214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.195:49481)
0927 15:10:00.042756 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:00.042820 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:00.044113 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:00.044725 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:00.044733 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:00.045260 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:10:00.045851 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:00.045869 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.046653 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.046656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:00.047641 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:00.047653 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.048140 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.048143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:00.048326 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:00.048905 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:10:00.048924 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:10:00.050582 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:10:00.052406 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:00.052413 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:00.052429 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:00.052729 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:00.053077 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:00.053080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:00.053081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:00.053284 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:00.053791 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:10:00.053798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:10:00.054085 (+ 287us) client_negotiation.cc:769] Sending connection context
0927 15:10:00.054285 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 15:10:00.054592 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":134,"threads_started":1}
I20260927 15:10:00.056885 21459 heartbeater.cc:347] Connected to a master server at 127.7.17.254:44411
I20260927 15:10:00.057147 21459 heartbeater.cc:464] Registering TS with master...
I20260927 15:10:00.057753 21459 heartbeater.cc:511] Master 127.7.17.254:44411 requested a full tablet report, sending...
I20260927 15:10:00.058861 20996 ts_manager.cc:194] Registered new tserver with Master: 8fac9d4b8f204372b80178dac5aa38cd (127.7.17.195:42987)
I20260927 15:10:00.059413 20996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:49481
I20260927 15:10:00.070148 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:10:00.079455 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:00.072123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33098 (local address 127.7.17.254:44411)
0927 15:10:00.072265 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:00.072268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:00.072422 (+ 154us) server_negotiation.cc:407] Connection header received
0927 15:10:00.072593 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:00.072596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:00.072649 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:00.072714 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:00.073451 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.073963 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.074768 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.074981 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.075068 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:00.076403 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:00.076407 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:00.076410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:00.076438 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:00.077929 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:00.078301 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:00.078303 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:00.078304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:00.078378 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:00.078610 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:00.078614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:00.078615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:00.078864 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:00.078974 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:00.079203 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 15:10:00.079331 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:10:00.082584 20996 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:10:00.085915 20996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:10:00.101445 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:00.095921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:42987 (local address 127.0.0.1:60594)
0927 15:10:00.096476 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:00.096487 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:00.096602 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:00.097494 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:00.097497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:00.097516 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:00.097618 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:00.097624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.098939 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.098942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:00.099996 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:00.100006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.100869 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.100873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:00.101002 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:00.101043 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 15:10:00.101154 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 15:10:00.101291 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":52,"thread_start_us":104,"threads_started":1}
I20260927 15:10:00.101455 21475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:00.096373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60594 (local address 127.7.17.195:42987)
0927 15:10:00.096806 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:00.096817 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:00.096867 (+ 50us) server_negotiation.cc:407] Connection header received
0927 15:10:00.096946 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:00.096952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:00.097163 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:00.097320 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:00.097745 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.098803 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.100122 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.100728 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.100852 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:00.101022 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:00.101195 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 15:10:00.101308 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":132,"threads_started":1}
I20260927 15:10:00.101989 21473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:00.095284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:42553 (local address 127.0.0.1:56038)
0927 15:10:00.096157 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:00.096169 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:00.096313 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:00.097298 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:00.097301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:00.097317 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:00.097403 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:00.097408 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.098414 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.098416 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:00.099987 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:00.099996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.101677 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.101680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:00.101753 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:00.101774 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:10:00.101822 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 15:10:00.101880 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":67,"thread_start_us":99,"threads_started":1}
I20260927 15:10:00.103469 21474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:00.096384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56038 (local address 127.7.17.194:42553)
0927 15:10:00.096794 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:00.096806 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:00.096868 (+ 62us) server_negotiation.cc:407] Connection header received
0927 15:10:00.096946 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:00.096952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:00.097163 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:00.097320 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:00.097483 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.098282 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.100944 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.101560 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.101680 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:00.101828 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:00.101882 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 15:10:00.101981 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":95,"threads_started":1}
I20260927 15:10:00.103689 21470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:00.095168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:43849 (local address 127.0.0.1:50238)
0927 15:10:00.095731 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:00.095786 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:00.095926 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:00.097079 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:00.097085 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:00.097107 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:00.097254 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:00.097270 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.098644 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.098647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:00.099984 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:00.099996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.103290 (+ 3294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.103294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:00.103382 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:00.103404 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:10:00.103480 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 15:10:00.103555 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":78,"thread_start_us":100,"threads_started":1}
I20260927 15:10:00.103962 21472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:00.095362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50238 (local address 127.7.17.193:43849)
0927 15:10:00.095978 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:00.095994 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:00.096019 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:10:00.096097 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:00.096102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:00.096926 (+ 824us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:00.097079 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:00.097396 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.098512 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.102222 (+ 3710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.103171 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.103262 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:00.103374 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:00.103541 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 15:10:00.103756 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":130,"threads_started":1}
I20260927 15:10:00.105218 21393 tablet_service.cc:1467] Processing CreateTablet for tablet eef46488164b4d24b60ab38b76cbe1d7 (DEFAULT_TABLE table=test-table [id=1b7ad76083c24ca3b2cfde702b5fbbe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:10:00.106253 21393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef46488164b4d24b60ab38b76cbe1d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:10:00.106541 21125 tablet_service.cc:1467] Processing CreateTablet for tablet eef46488164b4d24b60ab38b76cbe1d7 (DEFAULT_TABLE table=test-table [id=1b7ad76083c24ca3b2cfde702b5fbbe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:10:00.106410 21259 tablet_service.cc:1467] Processing CreateTablet for tablet eef46488164b4d24b60ab38b76cbe1d7 (DEFAULT_TABLE table=test-table [id=1b7ad76083c24ca3b2cfde702b5fbbe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:10:00.107591 21259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef46488164b4d24b60ab38b76cbe1d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:10:00.107591 21125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef46488164b4d24b60ab38b76cbe1d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:10:00.112988 21476 tablet_bootstrap.cc:492] T eef46488164b4d24b60ab38b76cbe1d7 P 8fac9d4b8f204372b80178dac5aa38cd: Bootstrap starting.
I20260927 15:10:00.115162 21477 tablet_bootstrap.cc:492] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff: Bootstrap starting.
I20260927 15:10:00.115725 21476 tablet_bootstrap.cc:654] T eef46488164b4d24b60ab38b76cbe1d7 P 8fac9d4b8f204372b80178dac5aa38cd: Neither blocks nor log segments found. Creating new log.
I20260927 15:10:00.116169 21478 tablet_bootstrap.cc:492] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095: Bootstrap starting.
I20260927 15:10:00.116901 21476 log.cc:824] T eef46488164b4d24b60ab38b76cbe1d7 P 8fac9d4b8f204372b80178dac5aa38cd: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:00.117872 21477 tablet_bootstrap.cc:654] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff: Neither blocks nor log segments found. Creating new log.
I20260927 15:10:00.118575 21478 tablet_bootstrap.cc:654] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095: Neither blocks nor log segments found. Creating new log.
I20260927 15:10:00.119191 21477 log.cc:824] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:00.119282 21476 tablet_bootstrap.cc:492] T eef46488164b4d24b60ab38b76cbe1d7 P 8fac9d4b8f204372b80178dac5aa38cd: No bootstrap required, opened a new log
I20260927 15:10:00.119292 21478 log.cc:824] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:00.119472 21476 ts_tablet_manager.cc:1397] T eef46488164b4d24b60ab38b76cbe1d7 P 8fac9d4b8f204372b80178dac5aa38cd: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260927 15:10:00.121176 21477 tablet_bootstrap.cc:492] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff: No bootstrap required, opened a new log
I20260927 15:10:00.121392 21477 ts_tablet_manager.cc:1397] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:10:00.121474 21478 tablet_bootstrap.cc:492] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095: No bootstrap required, opened a new log
I20260927 15:10:00.121662 21478 ts_tablet_manager.cc:1397] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260927 15:10:00.123448 21476 raft_consensus.cc:348] T eef46488164b4d24b60ab38b76cbe1d7 P 8fac9d4b8f204372b80178dac5aa38cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } } peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } }
I20260927 15:10:00.123757 21476 raft_consensus.cc:374] T eef46488164b4d24b60ab38b76cbe1d7 P 8fac9d4b8f204372b80178dac5aa38cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:10:00.123859 21476 raft_consensus.cc:729] T eef46488164b4d24b60ab38b76cbe1d7 P 8fac9d4b8f204372b80178dac5aa38cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fac9d4b8f204372b80178dac5aa38cd, State: Initialized, Role: FOLLOWER
I20260927 15:10:00.124396 21476 consensus_queue.cc:260] T eef46488164b4d24b60ab38b76cbe1d7 P 8fac9d4b8f204372b80178dac5aa38cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } } peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } }
I20260927 15:10:00.124356 21477 raft_consensus.cc:348] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } } peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } }
I20260927 15:10:00.124641 21477 raft_consensus.cc:374] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:10:00.124727 21477 raft_consensus.cc:729] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881cc60f613d4b408db7a4c7189ef7ff, State: Initialized, Role: FOLLOWER
I20260927 15:10:00.125262 21477 consensus_queue.cc:260] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } } peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } }
I20260927 15:10:00.125356 21478 raft_consensus.cc:348] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } } peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } }
I20260927 15:10:00.125680 21478 raft_consensus.cc:374] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:10:00.125700 21476 ts_tablet_manager.cc:1428] T eef46488164b4d24b60ab38b76cbe1d7 P 8fac9d4b8f204372b80178dac5aa38cd: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260927 15:10:00.125785 21478 raft_consensus.cc:729] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f73c2d3aac64266bf45eee00a10c095, State: Initialized, Role: FOLLOWER
I20260927 15:10:00.125967 21191 heartbeater.cc:503] Master 127.7.17.254:44411 was elected leader, sending a full tablet report...
I20260927 15:10:00.126027 21459 heartbeater.cc:503] Master 127.7.17.254:44411 was elected leader, sending a full tablet report...
I20260927 15:10:00.126246 21478 consensus_queue.cc:260] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } } peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } }
I20260927 15:10:00.127013 21325 heartbeater.cc:503] Master 127.7.17.254:44411 was elected leader, sending a full tablet report...
I20260927 15:10:00.127220 21478 ts_tablet_manager.cc:1428] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:10:00.127825 21477 ts_tablet_manager.cc:1428] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:10:00.264798 21484 raft_consensus.cc:482] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:10:00.265048 21484 raft_consensus.cc:504] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } } peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } }
I20260927 15:10:00.266244 21484 leader_election.cc:288] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 881cc60f613d4b408db7a4c7189ef7ff (127.7.17.193:43849), 8fac9d4b8f204372b80178dac5aa38cd (127.7.17.195:42987)
I20260927 15:10:00.269985 21472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:00.266578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:42083 (local address 127.7.17.193:43849)
0927 15:10:00.266704 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:00.266708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:00.266818 (+ 110us) server_negotiation.cc:407] Connection header received
0927 15:10:00.266934 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:00.266936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:00.266985 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:00.267062 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:00.267335 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.267907 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.268984 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.269406 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.269530 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:00.269590 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:00.269762 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 15:10:00.269847 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:10:00.269985 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:00.266493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:43849 (local address 127.7.17.194:42083)
0927 15:10:00.266702 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:00.266717 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:00.266824 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:00.267090 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:00.267093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:00.267111 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:00.267229 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:00.267234 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.268027 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.268031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:00.268862 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:00.268873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.269530 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.269534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:00.269644 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:00.269668 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:10:00.269735 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 15:10:00.269843 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":1}
I20260927 15:10:00.270102 21475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:00.266626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:56747 (local address 127.7.17.195:42987)
0927 15:10:00.266763 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:00.266767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:00.267003 (+ 236us) server_negotiation.cc:407] Connection header received
0927 15:10:00.267087 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:00.267091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:00.267136 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:00.267196 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:00.267459 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.267963 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.269039 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.269463 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.269578 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:00.269633 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:00.269806 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 15:10:00.269986 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:10:00.270150 21485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:00.266536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:42987 (local address 127.7.17.194:56747)
0927 15:10:00.266905 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:00.266917 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:00.267006 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:00.267230 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:00.267233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:00.267248 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:00.267353 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:00.267358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.268072 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.268075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:00.268917 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:00.268927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.269578 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.269582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:00.269689 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:00.269708 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 15:10:00.269772 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 15:10:00.269962 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":96,"thread_start_us":132,"threads_started":1}
I20260927 15:10:00.270821 21145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef46488164b4d24b60ab38b76cbe1d7" candidate_uuid: "8f73c2d3aac64266bf45eee00a10c095" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881cc60f613d4b408db7a4c7189ef7ff" is_pre_election: true
I20260927 15:10:00.270823 21413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef46488164b4d24b60ab38b76cbe1d7" candidate_uuid: "8f73c2d3aac64266bf45eee00a10c095" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fac9d4b8f204372b80178dac5aa38cd" is_pre_election: true
I20260927 15:10:00.271124 21413 raft_consensus.cc:2454] T eef46488164b4d24b60ab38b76cbe1d7 P 8fac9d4b8f204372b80178dac5aa38cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f73c2d3aac64266bf45eee00a10c095 in term 0.
I20260927 15:10:00.271126 21145 raft_consensus.cc:2454] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f73c2d3aac64266bf45eee00a10c095 in term 0.
I20260927 15:10:00.271624 21214 leader_election.cc:302] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 881cc60f613d4b408db7a4c7189ef7ff, 8f73c2d3aac64266bf45eee00a10c095, 8fac9d4b8f204372b80178dac5aa38cd; no voters:
I20260927 15:10:00.271917 21484 raft_consensus.cc:2781] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:10:00.272003 21484 raft_consensus.cc:482] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:10:00.272038 21484 raft_consensus.cc:3037] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:10:00.273133 21484 raft_consensus.cc:504] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } } peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } }
I20260927 15:10:00.273525 21484 leader_election.cc:288] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095 [CANDIDATE]: Term 1 election: Requested vote from peers 881cc60f613d4b408db7a4c7189ef7ff (127.7.17.193:43849), 8fac9d4b8f204372b80178dac5aa38cd (127.7.17.195:42987)
I20260927 15:10:00.273936 21413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef46488164b4d24b60ab38b76cbe1d7" candidate_uuid: "8f73c2d3aac64266bf45eee00a10c095" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fac9d4b8f204372b80178dac5aa38cd"
I20260927 15:10:00.273936 21145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef46488164b4d24b60ab38b76cbe1d7" candidate_uuid: "8f73c2d3aac64266bf45eee00a10c095" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881cc60f613d4b408db7a4c7189ef7ff"
I20260927 15:10:00.274078 21145 raft_consensus.cc:3037] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:10:00.274080 21413 raft_consensus.cc:3037] T eef46488164b4d24b60ab38b76cbe1d7 P 8fac9d4b8f204372b80178dac5aa38cd [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:10:00.275406 21413 raft_consensus.cc:2454] T eef46488164b4d24b60ab38b76cbe1d7 P 8fac9d4b8f204372b80178dac5aa38cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f73c2d3aac64266bf45eee00a10c095 in term 1.
I20260927 15:10:00.275399 21145 raft_consensus.cc:2454] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f73c2d3aac64266bf45eee00a10c095 in term 1.
I20260927 15:10:00.275755 21213 leader_election.cc:302] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 881cc60f613d4b408db7a4c7189ef7ff, 8f73c2d3aac64266bf45eee00a10c095; no voters:
I20260927 15:10:00.275985 21484 raft_consensus.cc:2781] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:10:00.276226 21484 raft_consensus.cc:686] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095 [term 1 LEADER]: Becoming Leader. State: Replica: 8f73c2d3aac64266bf45eee00a10c095, State: Running, Role: LEADER
I20260927 15:10:00.276602 21484 consensus_queue.cc:237] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } } peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } }
I20260927 15:10:00.280112 20998 catalog_manager.cc:5495] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f73c2d3aac64266bf45eee00a10c095 (127.7.17.194). New cstate: current_term: 1 leader_uuid: "8f73c2d3aac64266bf45eee00a10c095" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } health_report { overall_health: UNKNOWN } } }
W20260927 15:10:00.290840 21460 tablet.cc:2369] T eef46488164b4d24b60ab38b76cbe1d7 P 8fac9d4b8f204372b80178dac5aa38cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:10:00.291946 21474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:00.288089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56050 (local address 127.7.17.194:42553)
0927 15:10:00.288234 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:00.288238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:00.288252 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:10:00.288330 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:00.288332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:00.288388 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:00.288485 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:10:00.288699 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.289259 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.290221 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.290442 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.290534 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:00.290977 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:10:00.291491 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:10:00.291596 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:00.291690 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 15:10:00.291749 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:10:00.298725 21145 raft_consensus.cc:1261] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Refusing update from remote peer 8f73c2d3aac64266bf45eee00a10c095: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:10:00.298725 21413 raft_consensus.cc:1261] T eef46488164b4d24b60ab38b76cbe1d7 P 8fac9d4b8f204372b80178dac5aa38cd [term 1 FOLLOWER]: Refusing update from remote peer 8f73c2d3aac64266bf45eee00a10c095: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:10:00.299479 21486 consensus_queue.cc:1035] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095 [LEADER]: Connected to new peer: Peer: permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:10:00.299800 21484 consensus_queue.cc:1035] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:10:00.308035 21491 mvcc.cc:205] Tried to move back new op lower bound from 7333977294013718528 to 7333977293936189440. Current Snapshot: MvccSnapshot[applied={T|T < 7333977294013718528}]
I20260927 15:10:00.310981 21490 mvcc.cc:205] Tried to move back new op lower bound from 7333977294013718528 to 7333977293936189440. Current Snapshot: MvccSnapshot[applied={T|T < 7333977294013718528 or (T in {7333977294013718528})}]
I20260927 15:10:00.319271 21000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:10:00.319449 21000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:10:00.322202 21000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4: Sending DeleteTablet for 3 replicas of tablet eef46488164b4d24b60ab38b76cbe1d7
I20260927 15:10:00.323056 21125 tablet_service.cc:1514] Processing DeleteTablet for tablet eef46488164b4d24b60ab38b76cbe1d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:10:00 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:50238
I20260927 15:10:00.323601 21499 tablet_replica.cc:330] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff: stopping tablet replica
I20260927 15:10:00.323629 21393 tablet_service.cc:1514] Processing DeleteTablet for tablet eef46488164b4d24b60ab38b76cbe1d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:10:00 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:60594
I20260927 15:10:00.324055 21499 raft_consensus.cc:2229] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:10:00.324482 21499 raft_consensus.cc:2258] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:10:00.324525 21259 tablet_service.cc:1514] Processing DeleteTablet for tablet eef46488164b4d24b60ab38b76cbe1d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:10:00 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:56038
I20260927 15:10:00.324985 21501 tablet_replica.cc:330] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095: stopping tablet replica
I20260927 15:10:00.325337 21501 raft_consensus.cc:2229] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:10:00.325665 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 21062
I20260927 15:10:00.326797 21501 raft_consensus.cc:2258] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:10:00.328437 21501 ts_tablet_manager.cc:1905] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:10:00.331318 21501 ts_tablet_manager.cc:1918] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:10:00.331476 21501 log.cc:1196] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/wal/wals/eef46488164b4d24b60ab38b76cbe1d7
I20260927 15:10:00.331799 21501 ts_tablet_manager.cc:1939] T eef46488164b4d24b60ab38b76cbe1d7 P 8f73c2d3aac64266bf45eee00a10c095: Deleting consensus metadata
I20260927 15:10:00.331979 21500 tablet_replica.cc:330] T eef46488164b4d24b60ab38b76cbe1d7 P 8fac9d4b8f204372b80178dac5aa38cd: stopping tablet replica
I20260927 15:10:00.332369 21500 raft_consensus.cc:2229] T eef46488164b4d24b60ab38b76cbe1d7 P 8fac9d4b8f204372b80178dac5aa38cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:10:00.332736 21500 raft_consensus.cc:2258] T eef46488164b4d24b60ab38b76cbe1d7 P 8fac9d4b8f204372b80178dac5aa38cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:10:00.333105 20987 catalog_manager.cc:4841] TS 8f73c2d3aac64266bf45eee00a10c095 (127.7.17.194:42553): tablet eef46488164b4d24b60ab38b76cbe1d7 (table test-table [id=1b7ad76083c24ca3b2cfde702b5fbbe1]) successfully deleted
W20260927 15:10:00.333940 20985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.17.193:43849 (error 108)
I20260927 15:10:00.334178 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 21196
I20260927 15:10:00.334383 21500 ts_tablet_manager.cc:1905] T eef46488164b4d24b60ab38b76cbe1d7 P 8fac9d4b8f204372b80178dac5aa38cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:10:00.336701 21470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:00.336437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:43849 (local address 127.0.0.1:50248)
0927 15:10:00.336561 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:00.336627 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.193:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 15:10:00.336920 20985 catalog_manager.cc:4584] TS 881cc60f613d4b408db7a4c7189ef7ff (127.7.17.193:43849): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eef46488164b4d24b60ab38b76cbe1d7: Network error: Client connection negotiation failed: client connection to 127.7.17.193:43849: connect: Connection refused (error 111)
I20260927 15:10:00.337242 21500 ts_tablet_manager.cc:1918] T eef46488164b4d24b60ab38b76cbe1d7 P 8fac9d4b8f204372b80178dac5aa38cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:10:00.337347 21500 log.cc:1196] T eef46488164b4d24b60ab38b76cbe1d7 P 8fac9d4b8f204372b80178dac5aa38cd: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/eef46488164b4d24b60ab38b76cbe1d7
I20260927 15:10:00.337663 21500 ts_tablet_manager.cc:1939] T eef46488164b4d24b60ab38b76cbe1d7 P 8fac9d4b8f204372b80178dac5aa38cd: Deleting consensus metadata
I20260927 15:10:00.338935 20986 catalog_manager.cc:4841] TS 8fac9d4b8f204372b80178dac5aa38cd (127.7.17.195:42987): tablet eef46488164b4d24b60ab38b76cbe1d7 (table test-table [id=1b7ad76083c24ca3b2cfde702b5fbbe1]) successfully deleted
I20260927 15:10:00.340533 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 21330
I20260927 15:10:00.345537 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 20968
I20260927 15:10:00.351207 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:44411
--webserver_interface=127.7.17.254
--webserver_port=43913
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:44411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:10:00.466024 21503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:00.466339 21503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:00.466398 21503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:00.470739 21503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:10:00.470820 21503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:00.470840 21503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:00.470858 21503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:10:00.470875 21503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:10:00.476584 21503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:44411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:44411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=43913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:10:00.477780 21503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:00.478847 21503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:10:00.492004 21503 server_base.cc:1034] running on GCE node
W20260927 15:10:00.491925 21510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:00.491887 21509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:00.491895 21512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:00.492753 21503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:00.494071 21503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:00.495263 21503 hybrid_clock.cc:648] HybridClock initialized: now 1790521800495235 us; error 42 us; skew 500 ppm
Sep 27 15:10:00 dist-test-slave-hhpq krb5kdc[20955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521800, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:00.498302 21503 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:10:00.499337 21503 webserver.cc:460] Webserver started at http://127.7.17.254:43913/ using document root <none> and password file <none>
I20260927 15:10:00.499943 21503 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:00.500027 21503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:00.504308 21503 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 15:10:00.506748 21519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:00.508255 21503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:10:00.508401 21503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "f13f064244294db4b21bd6bf73cf2da4"
format_stamp: "Formatted at 2026-09-27 15:09:59 on dist-test-slave-hhpq"
server_key: "a92b9600e0e9d7a7d7b1c1183e037136"
server_key_iv: "c4743d41bcfb1755983e65f0d82596fb"
server_key_version: "encryptionkey@0"
I20260927 15:10:00.509095 21503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:00.534192 21503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:00.537722 21503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:00.537914 21503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:00.544862 21503 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:44411
I20260927 15:10:00.545136 21571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:44411 every 8 connection(s)
I20260927 15:10:00.546012 21503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:10:00.547951 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 21503
I20260927 15:10:00.548355 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:43849
--local_ip_for_outbound_sockets=127.7.17.193
--tserver_master_addrs=127.7.17.254:44411
--webserver_port=34993
--webserver_interface=127.7.17.193
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:10:00.550690 21572 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:10:00.552501 21572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4: Bootstrap starting.
I20260927 15:10:00.570068 21572 log.cc:824] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:00.579559 21572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:10:00.579964 21572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4: Bootstrap complete.
I20260927 15:10:00.584033 21572 raft_consensus.cc:348] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13f064244294db4b21bd6bf73cf2da4" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 44411 } }
I20260927 15:10:00.584461 21572 raft_consensus.cc:729] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f13f064244294db4b21bd6bf73cf2da4, State: Initialized, Role: FOLLOWER
I20260927 15:10:00.585017 21572 consensus_queue.cc:260] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13f064244294db4b21bd6bf73cf2da4" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 44411 } }
I20260927 15:10:00.585148 21572 raft_consensus.cc:388] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:10:00.585220 21572 raft_consensus.cc:482] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:10:00.585318 21572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:10:00.587431 21572 raft_consensus.cc:504] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13f064244294db4b21bd6bf73cf2da4" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 44411 } }
I20260927 15:10:00.587818 21572 leader_election.cc:302] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f13f064244294db4b21bd6bf73cf2da4; no voters:
I20260927 15:10:00.588147 21572 leader_election.cc:288] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:10:00.588255 21576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:10:00.588478 21576 raft_consensus.cc:686] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 2 LEADER]: Becoming Leader. State: Replica: f13f064244294db4b21bd6bf73cf2da4, State: Running, Role: LEADER
I20260927 15:10:00.588904 21576 consensus_queue.cc:237] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13f064244294db4b21bd6bf73cf2da4" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 44411 } }
I20260927 15:10:00.589542 21572 sys_catalog.cc:564] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:10:00.590456 21577 sys_catalog.cc:455] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f13f064244294db4b21bd6bf73cf2da4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13f064244294db4b21bd6bf73cf2da4" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 44411 } } }
I20260927 15:10:00.590502 21578 sys_catalog.cc:455] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f13f064244294db4b21bd6bf73cf2da4. Latest consensus state: current_term: 2 leader_uuid: "f13f064244294db4b21bd6bf73cf2da4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13f064244294db4b21bd6bf73cf2da4" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 44411 } } }
I20260927 15:10:00.590588 21577 sys_catalog.cc:458] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [sys.catalog]: This master's current role is: LEADER
I20260927 15:10:00.590613 21578 sys_catalog.cc:458] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [sys.catalog]: This master's current role is: LEADER
I20260927 15:10:00.590965 21585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:10:00.595880 21585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:10:00.596576 21585 catalog_manager.cc:1259] Loaded cluster ID: 313a2279e2cd4e638c59a691842e4f68
I20260927 15:10:00.596655 21585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:10:00.597810 21585 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 15:10:00.600545 21585 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 15:10:00.605839 21585 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:10:00.606676 21585 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4: Loaded TSK: 0
I20260927 15:10:00.607694 21585 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:10:00.675055 21574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:00.675355 21574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:00.675418 21574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:00.679577 21574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:00.679663 21574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:00.679759 21574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:10:00.685323 21574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:43849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=34993
--tserver_master_addrs=127.7.17.254:44411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:10:00.686431 21574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:00.687502 21574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:00.698766 21600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:00.698823 21574 server_base.cc:1034] running on GCE node
W20260927 15:10:00.698766 21599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:00.699095 21602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:00.699628 21574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:00.700327 21574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:00.701505 21574 hybrid_clock.cc:648] HybridClock initialized: now 1790521800701489 us; error 29 us; skew 500 ppm
Sep 27 15:10:00 dist-test-slave-hhpq krb5kdc[20955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521800, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:00.704492 21574 init.cc:377] Logged in from keytab as kudu/127.7.17.193@KRBTEST.COM (short username kudu)
I20260927 15:10:00.705538 21574 webserver.cc:460] Webserver started at http://127.7.17.193:34993/ using document root <none> and password file <none>
I20260927 15:10:00.706130 21574 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:00.706221 21574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:00.710733 21574 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:10:00.713240 21609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:00.714690 21574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:10:00.714804 21574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "881cc60f613d4b408db7a4c7189ef7ff"
format_stamp: "Formatted at 2026-09-27 15:09:59 on dist-test-slave-hhpq"
server_key: "53a2aa6ca8989f8136f80a8d47d45b2d"
server_key_iv: "b024aea45856006327588994891f1540"
server_key_version: "encryptionkey@0"
I20260927 15:10:00.715406 21574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:00.738021 21574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:00.741616 21574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:00.741803 21574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:00.742425 21574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:00.743552 21616 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 15:10:00.747848 21574 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 15:10:00.747934 21574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 15:10:00.748005 21574 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 15:10:00.749847 21574 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 15:10:00.749904 21574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260927 15:10:00.750257 21616 tablet_bootstrap.cc:492] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff: Bootstrap starting.
I20260927 15:10:00.759649 21574 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:43849
I20260927 15:10:00.759703 21723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:43849 every 8 connection(s)
I20260927 15:10:00.760649 21574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:10:00.765394 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 21574
I20260927 15:10:00.765822 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:42553
--local_ip_for_outbound_sockets=127.7.17.194
--tserver_master_addrs=127.7.17.254:44411
--webserver_port=40863
--webserver_interface=127.7.17.194
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:10:00.769627 21616 log.cc:824] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:10:00 dist-test-slave-hhpq krb5kdc[20955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521800, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:10:00.778070 21726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:00.763032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:34729 (local address 127.7.17.254:44411)
0927 15:10:00.763451 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:00.763472 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:00.763802 (+ 330us) server_negotiation.cc:407] Connection header received
0927 15:10:00.765028 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:00.765046 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:00.765596 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:00.766051 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:00.767647 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.768644 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.769903 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.770661 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.770824 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:00.773294 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:00.773299 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:00.773312 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:00.773343 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:00.774928 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:00.775425 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:00.775430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:00.775434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:00.775552 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:00.775871 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:00.775874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:00.775875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:00.776229 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:00.776357 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:00.776917 (+ 560us) server_negotiation.cc:299] Negotiation successful
0927 15:10:00.777178 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":131,"threads_started":1}
I20260927 15:10:00.778083 21727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:00.762967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.193:34729)
0927 15:10:00.763612 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:00.763668 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:00.764526 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:00.766327 (+ 1801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:00.766338 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:00.766823 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:10:00.767464 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:00.767482 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.768814 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.768817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:00.769764 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:00.769777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.770805 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.770810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:00.771004 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:00.771603 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:10:00.771620 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:10:00.773094 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:10:00.775043 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:00.775049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:00.775063 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:00.775317 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:00.775640 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:00.775643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:00.775645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:00.775775 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:00.776350 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:10:00.776356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:10:00.776599 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 15:10:00.776759 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 15:10:00.777140 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":127,"threads_started":1}
I20260927 15:10:00.779028 21616 tablet_bootstrap.cc:492] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:10:00.779467 21616 tablet_bootstrap.cc:492] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff: Bootstrap complete.
I20260927 15:10:00.780604 21724 heartbeater.cc:347] Connected to a master server at 127.7.17.254:44411
I20260927 15:10:00.780864 21616 ts_tablet_manager.cc:1397] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff: Time spent bootstrapping tablet: real 0.031s user 0.026s sys 0.004s
I20260927 15:10:00.780865 21724 heartbeater.cc:464] Registering TS with master...
I20260927 15:10:00.781445 21724 heartbeater.cc:511] Master 127.7.17.254:44411 requested a full tablet report, sending...
I20260927 15:10:00.783854 21531 ts_manager.cc:194] Registered new tserver with Master: 881cc60f613d4b408db7a4c7189ef7ff (127.7.17.193:43849)
I20260927 15:10:00.783932 21616 raft_consensus.cc:348] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } } peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } }
I20260927 15:10:00.784513 21616 raft_consensus.cc:729] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881cc60f613d4b408db7a4c7189ef7ff, State: Initialized, Role: FOLLOWER
I20260927 15:10:00.785130 21616 consensus_queue.cc:260] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } } peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } }
I20260927 15:10:00.786286 21616 ts_tablet_manager.cc:1428] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 15:10:00.788054 21531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:34729
I20260927 15:10:00.791013 21724 heartbeater.cc:503] Master 127.7.17.254:44411 was elected leader, sending a full tablet report...
Sep 27 15:10:00 dist-test-slave-hhpq krb5kdc[20955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521800, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for kudu/127.7.17.193@KRBTEST.COM
I20260927 15:10:00.798408 21731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:00.787059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:43849 (local address 127.0.0.1:50264)
0927 15:10:00.787551 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:00.787609 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:00.787782 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:00.788582 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:00.788588 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:00.789125 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:10:00.789319 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:00.789329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.790623 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.790627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:00.791485 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:00.791498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.792213 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.792217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:00.792353 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:00.793102 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:10:00.793114 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:10:00.794552 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:10:00.796383 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:00.796394 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:00.796400 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:00.796704 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:00.797070 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:00.797073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:00.797075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:00.797174 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:00.797642 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:10:00.797648 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:10:00.797809 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 15:10:00.797966 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 15:10:00.798206 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":151,"threads_started":1}
I20260927 15:10:00.798444 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:00.787134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50264 (local address 127.7.17.193:43849)
0927 15:10:00.788077 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:00.788089 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:00.788112 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:10:00.788186 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:00.788192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:00.788418 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:00.788549 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:00.789500 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.790510 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.791614 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.792087 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.792199 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:00.794702 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:00.794711 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:00.794722 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:00.794746 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:00.796261 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:00.796888 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:00.796895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:00.796896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:00.796987 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:00.797264 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:00.797266 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:00.797268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:00.797536 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:00.797641 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:00.798144 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 15:10:00.798310 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":96,"threads_started":1}
I20260927 15:10:00.799743 21658 tablet_service.cc:1514] Processing DeleteTablet for tablet eef46488164b4d24b60ab38b76cbe1d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:10:00 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:50264
I20260927 15:10:00.800611 21733 tablet_replica.cc:330] stopping tablet replica
I20260927 15:10:00.800981 21733 raft_consensus.cc:2229] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:10:00.801244 21733 raft_consensus.cc:2258] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:10:00.802212 21733 ts_tablet_manager.cc:1905] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:10:00.806106 21733 ts_tablet_manager.cc:1918] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:10:00.806213 21733 log.cc:1196] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/wal/wals/eef46488164b4d24b60ab38b76cbe1d7
I20260927 15:10:00.806443 21733 ts_tablet_manager.cc:1939] T eef46488164b4d24b60ab38b76cbe1d7 P 881cc60f613d4b408db7a4c7189ef7ff: Deleting consensus metadata
I20260927 15:10:00.808360 21521 catalog_manager.cc:4841] TS 881cc60f613d4b408db7a4c7189ef7ff (127.7.17.193:43849): tablet eef46488164b4d24b60ab38b76cbe1d7 (table test-table [id=1b7ad76083c24ca3b2cfde702b5fbbe1]) successfully deleted
W20260927 15:10:00.889601 21728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:00.889894 21728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:00.889957 21728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:00.894275 21728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:00.894348 21728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:00.894436 21728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:10:00.900023 21728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:42553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=40863
--tserver_master_addrs=127.7.17.254:44411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:10:00.901211 21728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:00.902292 21728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:00.912438 21741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:00.912493 21728 server_base.cc:1034] running on GCE node
W20260927 15:10:00.912432 21739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:00.912461 21738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:00.913291 21728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:00.914069 21728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:00.915251 21728 hybrid_clock.cc:648] HybridClock initialized: now 1790521800915238 us; error 31 us; skew 500 ppm
Sep 27 15:10:00 dist-test-slave-hhpq krb5kdc[20955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521800, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:00.918195 21728 init.cc:377] Logged in from keytab as kudu/127.7.17.194@KRBTEST.COM (short username kudu)
I20260927 15:10:00.919255 21728 webserver.cc:460] Webserver started at http://127.7.17.194:40863/ using document root <none> and password file <none>
I20260927 15:10:00.919899 21728 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:00.919983 21728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:00.924242 21728 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:10:00.926527 21748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:00.927894 21728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:10:00.928010 21728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "8f73c2d3aac64266bf45eee00a10c095"
format_stamp: "Formatted at 2026-09-27 15:09:59 on dist-test-slave-hhpq"
server_key: "961d8a9a99c24315704f0c53399c3f39"
server_key_iv: "2937772d44511fc1862e8fc36057298a"
server_key_version: "encryptionkey@0"
I20260927 15:10:00.928557 21728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:00.951269 21728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:00.954758 21728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:00.954942 21728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:00.955546 21728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:00.956429 21728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:10:00.956477 21728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:00.956517 21728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:10:00.956537 21728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:00.965529 21728 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:42553
I20260927 15:10:00.965559 21861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:42553 every 8 connection(s)
I20260927 15:10:00.966594 21728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:10:00.974050 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 21728
I20260927 15:10:00.974452 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:42987
--local_ip_for_outbound_sockets=127.7.17.195
--tserver_master_addrs=127.7.17.254:44411
--webserver_port=46373
--webserver_interface=127.7.17.195
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:10:00 dist-test-slave-hhpq krb5kdc[20955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521800, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:10:00.980954 21726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:00.968548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:48051 (local address 127.7.17.254:44411)
0927 15:10:00.968688 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:00.968693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:00.969870 (+ 1177us) server_negotiation.cc:407] Connection header received
0927 15:10:00.970880 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:00.970884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:00.970950 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:00.971026 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:00.972347 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.972934 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.974119 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.974377 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.974473 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:00.977107 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:00.977110 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:00.977112 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:00.977141 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:00.978672 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:00.979269 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:00.979272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:00.979274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:00.979361 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:00.979718 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:00.979721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:00.979723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:00.979933 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:00.980022 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:00.980587 (+ 565us) server_negotiation.cc:299] Negotiation successful
0927 15:10:00.980740 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:10:00.981841 21864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:00.968994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.194:48051)
0927 15:10:00.969686 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:00.969741 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:00.970658 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:00.971244 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:00.971252 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:00.971638 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:10:00.972182 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:00.972197 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.973067 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.973070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:00.973990 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:00.974001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:00.974536 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:00.974540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:00.974715 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:00.975350 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:10:00.975368 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:10:00.976867 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:10:00.978813 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:00.978821 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:00.978842 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:00.979152 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:00.979451 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:00.979454 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:00.979456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:00.979622 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:00.980041 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:10:00.980048 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:10:00.980339 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 15:10:00.980527 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 15:10:00.980941 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":111,"threads_started":1}
I20260927 15:10:00.983228 21862 heartbeater.cc:347] Connected to a master server at 127.7.17.254:44411
I20260927 15:10:00.983501 21862 heartbeater.cc:464] Registering TS with master...
I20260927 15:10:00.984061 21862 heartbeater.cc:511] Master 127.7.17.254:44411 requested a full tablet report, sending...
I20260927 15:10:00.985242 21531 ts_manager.cc:194] Registered new tserver with Master: 8f73c2d3aac64266bf45eee00a10c095 (127.7.17.194:42553)
I20260927 15:10:00.985831 21531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:48051
W20260927 15:10:01.088019 21865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:01.088316 21865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:01.088378 21865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:01.092792 21865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:01.092900 21865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:01.092985 21865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:10:01.098635 21865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:42987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=46373
--tserver_master_addrs=127.7.17.254:44411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:10:01.099771 21865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:01.100940 21865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:01.111003 21870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:01.111063 21865 server_base.cc:1034] running on GCE node
W20260927 15:10:01.110992 21873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:01.110987 21871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:01.111567 21865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:01.112212 21865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:01.113371 21865 hybrid_clock.cc:648] HybridClock initialized: now 1790521801113331 us; error 41 us; skew 500 ppm
Sep 27 15:10:01 dist-test-slave-hhpq krb5kdc[20955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521801, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:01.116070 21865 init.cc:377] Logged in from keytab as kudu/127.7.17.195@KRBTEST.COM (short username kudu)
I20260927 15:10:01.117059 21865 webserver.cc:460] Webserver started at http://127.7.17.195:46373/ using document root <none> and password file <none>
I20260927 15:10:01.117631 21865 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:01.117715 21865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:01.121829 21865 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:10:01.123987 21880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:01.125432 21865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:10:01.125563 21865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "8fac9d4b8f204372b80178dac5aa38cd"
format_stamp: "Formatted at 2026-09-27 15:09:59 on dist-test-slave-hhpq"
server_key: "bf99e66287c79017e81df9bc48ab2d7c"
server_key_iv: "9201de33ac3b0045d787800f99775c6f"
server_key_version: "encryptionkey@0"
I20260927 15:10:01.126145 21865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:01.150915 21865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:01.154033 21865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:01.154219 21865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:01.154839 21865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:01.155709 21865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:10:01.155757 21865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:01.155812 21865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:10:01.155841 21865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:01.164613 21865 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:42987
I20260927 15:10:01.164640 21993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:42987 every 8 connection(s)
I20260927 15:10:01.165728 21865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:10:01.171582 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 21865
Sep 27 15:10:01 dist-test-slave-hhpq krb5kdc[20955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521801, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:10:01.179965 21726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:01.167565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:35603 (local address 127.7.17.254:44411)
0927 15:10:01.167715 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:01.167719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:01.168876 (+ 1157us) server_negotiation.cc:407] Connection header received
0927 15:10:01.169820 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:01.169823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:01.169873 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:01.169937 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:01.171209 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.171965 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.173290 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.173625 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.173736 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:01.176332 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:01.176343 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:01.176346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:01.176375 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:01.177862 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:01.178395 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:01.178399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:01.178401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:01.178480 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:01.178803 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:01.178806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:01.178808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:01.179004 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:01.179094 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:01.179650 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 15:10:01.179787 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47,"spinlock_wait_cycles":896}
I20260927 15:10:01.180079 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:01.172239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33104 (local address 127.7.17.254:44411)
0927 15:10:01.172473 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:01.172478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:01.172630 (+ 152us) server_negotiation.cc:407] Connection header received
0927 15:10:01.172769 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:01.172772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:01.172880 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:01.172956 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:01.173766 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.174425 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.175280 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.175507 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.175585 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:01.177120 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:01.177123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:01.177126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:01.177169 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:01.178510 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:01.178899 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:01.178902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:01.178903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:01.178975 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:01.179222 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:01.179225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:01.179227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:01.179408 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:01.179615 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:01.179753 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 15:10:01.179881 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20260927 15:10:01.181402 21996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:01.168033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.195:35603)
0927 15:10:01.168700 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:01.168743 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:01.169579 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:01.170143 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:01.170151 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:01.170540 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:10:01.171039 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:01.171052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.172094 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.172097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:01.173180 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:01.173190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.173775 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.173779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:01.173971 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:01.174558 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:10:01.174574 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:10:01.176150 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:10:01.177972 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:01.177978 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:01.177997 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:01.178291 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:01.178562 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:01.178565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:01.178567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:01.178717 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:01.179102 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:10:01.179109 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:10:01.179410 (+ 301us) client_negotiation.cc:769] Sending connection context
0927 15:10:01.179593 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 15:10:01.179960 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":117,"threads_started":1}
I20260927 15:10:01.182633 21994 heartbeater.cc:347] Connected to a master server at 127.7.17.254:44411
I20260927 15:10:01.182875 21994 heartbeater.cc:464] Registering TS with master...
I20260927 15:10:01.183391 21994 heartbeater.cc:511] Master 127.7.17.254:44411 requested a full tablet report, sending...
I20260927 15:10:01.184397 21531 ts_manager.cc:194] Registered new tserver with Master: 8fac9d4b8f204372b80178dac5aa38cd (127.7.17.195:42987)
I20260927 15:10:01.185091 21531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:35603
I20260927 15:10:01.194487 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:10:01.199120 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:01.195155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33120 (local address 127.7.17.254:44411)
0927 15:10:01.195295 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:01.195300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:01.195475 (+ 175us) server_negotiation.cc:407] Connection header received
0927 15:10:01.195664 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:01.195667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:01.195838 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:01.195938 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:10:01.196210 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.196751 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.197662 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.197903 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.197994 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:01.198314 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:10:01.198729 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:10:01.198853 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:01.198909 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 15:10:01.198979 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:10:01.199815 21531 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:33120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:10:01.203250 21531 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:10:01.212427 21658 tablet_service.cc:1467] Processing CreateTablet for tablet 60b457e6e9fd4028afe96c14de957b33 (DEFAULT_TABLE table=test-table [id=a4a97e060a7949bdba4642dfb076bf17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:10:01.213505 21658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60b457e6e9fd4028afe96c14de957b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:10:01.216682 21616 tablet_bootstrap.cc:492] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff: Bootstrap starting.
I20260927 15:10:01.217085 21731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:01.210723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:42553 (local address 127.0.0.1:56066)
0927 15:10:01.210842 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:01.210858 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:01.210946 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:01.211896 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:01.211899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:01.211919 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:01.212042 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:01.212049 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.213688 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.213692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:01.214999 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:01.215014 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.216545 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.216549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:01.216663 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:01.216740 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 15:10:01.216819 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 15:10:01.216949 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 15:10:01.217443 22003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:01.210996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56066 (local address 127.7.17.194:42553)
0927 15:10:01.211329 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:01.211342 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:01.211380 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:10:01.211471 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:01.211478 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:01.211714 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:01.212212 (+ 498us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:01.212410 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.213526 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.215371 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.216412 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.216524 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:01.216677 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:01.216966 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 15:10:01.217080 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":123,"threads_started":1}
I20260927 15:10:01.217653 21616 tablet_bootstrap.cc:654] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff: Neither blocks nor log segments found. Creating new log.
I20260927 15:10:01.220691 22004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:01.211132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:42987 (local address 127.0.0.1:60598)
0927 15:10:01.211409 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:01.211421 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:01.211501 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:01.212954 (+ 1453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:01.212957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:01.212977 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:01.213073 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:01.213079 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.216972 (+ 3893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.216976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:01.218079 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:01.218091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.220239 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.220242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:01.220362 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:01.220385 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:10:01.220459 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 15:10:01.220547 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":106,"threads_started":1}
I20260927 15:10:01.220726 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:01.211576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60598 (local address 127.7.17.195:42987)
0927 15:10:01.212225 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:01.212237 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:01.212260 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:10:01.212334 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:01.212339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:01.212558 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:01.215505 (+ 2947us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:01.215727 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.216794 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.219042 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.220107 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.220228 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:01.220388 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:01.220503 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 15:10:01.220600 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":165,"threads_started":1}
I20260927 15:10:01.221386 21796 tablet_service.cc:1467] Processing CreateTablet for tablet 60b457e6e9fd4028afe96c14de957b33 (DEFAULT_TABLE table=test-table [id=a4a97e060a7949bdba4642dfb076bf17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:10:01.221961 21616 tablet_bootstrap.cc:492] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff: No bootstrap required, opened a new log
I20260927 15:10:01.222057 21616 ts_tablet_manager.cc:1397] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 15:10:01.222574 21616 raft_consensus.cc:348] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } } peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } }
I20260927 15:10:01.222709 21616 raft_consensus.cc:374] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:10:01.222751 21616 raft_consensus.cc:729] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881cc60f613d4b408db7a4c7189ef7ff, State: Initialized, Role: FOLLOWER
I20260927 15:10:01.222882 21616 consensus_queue.cc:260] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } } peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } }
I20260927 15:10:01.223155 21616 ts_tablet_manager.cc:1428] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:10:01.223369 21796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60b457e6e9fd4028afe96c14de957b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:10:01.224593 21928 tablet_service.cc:1467] Processing CreateTablet for tablet 60b457e6e9fd4028afe96c14de957b33 (DEFAULT_TABLE table=test-table [id=a4a97e060a7949bdba4642dfb076bf17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:10:01.225711 21928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60b457e6e9fd4028afe96c14de957b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:10:01.231787 22007 tablet_bootstrap.cc:492] T 60b457e6e9fd4028afe96c14de957b33 P 8f73c2d3aac64266bf45eee00a10c095: Bootstrap starting.
I20260927 15:10:01.232682 22008 tablet_bootstrap.cc:492] T 60b457e6e9fd4028afe96c14de957b33 P 8fac9d4b8f204372b80178dac5aa38cd: Bootstrap starting.
I20260927 15:10:01.234547 22007 tablet_bootstrap.cc:654] T 60b457e6e9fd4028afe96c14de957b33 P 8f73c2d3aac64266bf45eee00a10c095: Neither blocks nor log segments found. Creating new log.
I20260927 15:10:01.235288 22007 log.cc:824] T 60b457e6e9fd4028afe96c14de957b33 P 8f73c2d3aac64266bf45eee00a10c095: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:01.235478 22008 tablet_bootstrap.cc:654] T 60b457e6e9fd4028afe96c14de957b33 P 8fac9d4b8f204372b80178dac5aa38cd: Neither blocks nor log segments found. Creating new log.
I20260927 15:10:01.236173 22008 log.cc:824] T 60b457e6e9fd4028afe96c14de957b33 P 8fac9d4b8f204372b80178dac5aa38cd: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:01.237398 22007 tablet_bootstrap.cc:492] T 60b457e6e9fd4028afe96c14de957b33 P 8f73c2d3aac64266bf45eee00a10c095: No bootstrap required, opened a new log
I20260927 15:10:01.237593 22007 ts_tablet_manager.cc:1397] T 60b457e6e9fd4028afe96c14de957b33 P 8f73c2d3aac64266bf45eee00a10c095: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260927 15:10:01.238049 22008 tablet_bootstrap.cc:492] T 60b457e6e9fd4028afe96c14de957b33 P 8fac9d4b8f204372b80178dac5aa38cd: No bootstrap required, opened a new log
I20260927 15:10:01.238221 22008 ts_tablet_manager.cc:1397] T 60b457e6e9fd4028afe96c14de957b33 P 8fac9d4b8f204372b80178dac5aa38cd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:10:01.240945 22007 raft_consensus.cc:348] T 60b457e6e9fd4028afe96c14de957b33 P 8f73c2d3aac64266bf45eee00a10c095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } } peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } }
I20260927 15:10:01.241200 22007 raft_consensus.cc:374] T 60b457e6e9fd4028afe96c14de957b33 P 8f73c2d3aac64266bf45eee00a10c095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:10:01.241273 22007 raft_consensus.cc:729] T 60b457e6e9fd4028afe96c14de957b33 P 8f73c2d3aac64266bf45eee00a10c095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f73c2d3aac64266bf45eee00a10c095, State: Initialized, Role: FOLLOWER
I20260927 15:10:01.241312 22008 raft_consensus.cc:348] T 60b457e6e9fd4028afe96c14de957b33 P 8fac9d4b8f204372b80178dac5aa38cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } } peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } }
I20260927 15:10:01.241597 22008 raft_consensus.cc:374] T 60b457e6e9fd4028afe96c14de957b33 P 8fac9d4b8f204372b80178dac5aa38cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:10:01.241657 22008 raft_consensus.cc:729] T 60b457e6e9fd4028afe96c14de957b33 P 8fac9d4b8f204372b80178dac5aa38cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fac9d4b8f204372b80178dac5aa38cd, State: Initialized, Role: FOLLOWER
I20260927 15:10:01.241751 22007 consensus_queue.cc:260] T 60b457e6e9fd4028afe96c14de957b33 P 8f73c2d3aac64266bf45eee00a10c095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } } peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } }
I20260927 15:10:01.242050 22008 consensus_queue.cc:260] T 60b457e6e9fd4028afe96c14de957b33 P 8fac9d4b8f204372b80178dac5aa38cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } } peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } }
I20260927 15:10:01.242471 21862 heartbeater.cc:503] Master 127.7.17.254:44411 was elected leader, sending a full tablet report...
I20260927 15:10:01.242755 22007 ts_tablet_manager.cc:1428] T 60b457e6e9fd4028afe96c14de957b33 P 8f73c2d3aac64266bf45eee00a10c095: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 15:10:01.242996 22008 ts_tablet_manager.cc:1428] T 60b457e6e9fd4028afe96c14de957b33 P 8fac9d4b8f204372b80178dac5aa38cd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 15:10:01.243155 21994 heartbeater.cc:503] Master 127.7.17.254:44411 was elected leader, sending a full tablet report...
W20260927 15:10:01.261910 21725 tablet.cc:2369] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:10:01.357467 21730 raft_consensus.cc:482] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:10:01.357705 21730 raft_consensus.cc:504] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } } peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } }
I20260927 15:10:01.358830 21730 leader_election.cc:288] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f73c2d3aac64266bf45eee00a10c095 (127.7.17.194:42553), 8fac9d4b8f204372b80178dac5aa38cd (127.7.17.195:42987)
I20260927 15:10:01.364364 22003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:01.359237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:51079 (local address 127.7.17.194:42553)
0927 15:10:01.359380 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:01.359384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:01.359569 (+ 185us) server_negotiation.cc:407] Connection header received
0927 15:10:01.359774 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:01.359798 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:01.359845 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:01.359937 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:01.360361 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.361252 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.362994 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.363642 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.363849 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:01.363891 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:01.364142 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 15:10:01.364223 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:10:01.364364 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:01.359250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:42307 (local address 127.7.17.195:42987)
0927 15:10:01.359465 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:01.359468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:01.359572 (+ 104us) server_negotiation.cc:407] Connection header received
0927 15:10:01.359774 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:01.359778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:01.359838 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:01.359937 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:01.360375 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.361239 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.362994 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.363683 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.363808 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:01.363868 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:01.364142 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 15:10:01.364223 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260927 15:10:01.364408 22014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:01.359109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:42987 (local address 127.7.17.193:42307)
0927 15:10:01.359499 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:01.359509 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:01.359635 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:01.359961 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:01.359964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:01.359992 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:01.360189 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:01.360197 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.361409 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.361413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:01.362852 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:01.362859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.363838 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.363842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:01.363964 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:01.364030 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 15:10:01.364112 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 15:10:01.364234 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":204,"spinlock_wait_cycles":32384,"thread_start_us":82,"threads_started":1}
I20260927 15:10:01.364408 22013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:01.359109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:42553 (local address 127.7.17.193:51079)
0927 15:10:01.359461 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:01.359477 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:01.359635 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:01.359961 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:01.359964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:01.359992 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:01.360196 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:01.360202 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.361432 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.361434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:01.362816 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:01.362830 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.363806 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.363810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:01.363945 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:01.364037 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 15:10:01.364112 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 15:10:01.364237 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":163,"spinlock_wait_cycles":13696,"thread_start_us":73,"threads_started":1}
I20260927 15:10:01.365307 21816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60b457e6e9fd4028afe96c14de957b33" candidate_uuid: "881cc60f613d4b408db7a4c7189ef7ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f73c2d3aac64266bf45eee00a10c095" is_pre_election: true
I20260927 15:10:01.365311 21948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60b457e6e9fd4028afe96c14de957b33" candidate_uuid: "881cc60f613d4b408db7a4c7189ef7ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fac9d4b8f204372b80178dac5aa38cd" is_pre_election: true
I20260927 15:10:01.365648 21948 raft_consensus.cc:2454] T 60b457e6e9fd4028afe96c14de957b33 P 8fac9d4b8f204372b80178dac5aa38cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 881cc60f613d4b408db7a4c7189ef7ff in term 0.
I20260927 15:10:01.365648 21816 raft_consensus.cc:2454] T 60b457e6e9fd4028afe96c14de957b33 P 8f73c2d3aac64266bf45eee00a10c095 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 881cc60f613d4b408db7a4c7189ef7ff in term 0.
I20260927 15:10:01.366163 21612 leader_election.cc:302] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 881cc60f613d4b408db7a4c7189ef7ff, 8fac9d4b8f204372b80178dac5aa38cd; no voters:
I20260927 15:10:01.366475 21730 raft_consensus.cc:2781] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:10:01.366573 21730 raft_consensus.cc:482] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:10:01.366605 21730 raft_consensus.cc:3037] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:10:01.367707 21730 raft_consensus.cc:504] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } } peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } }
I20260927 15:10:01.368122 21730 leader_election.cc:288] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 1 election: Requested vote from peers 8f73c2d3aac64266bf45eee00a10c095 (127.7.17.194:42553), 8fac9d4b8f204372b80178dac5aa38cd (127.7.17.195:42987)
I20260927 15:10:01.368525 21948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60b457e6e9fd4028afe96c14de957b33" candidate_uuid: "881cc60f613d4b408db7a4c7189ef7ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fac9d4b8f204372b80178dac5aa38cd"
I20260927 15:10:01.368577 21816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60b457e6e9fd4028afe96c14de957b33" candidate_uuid: "881cc60f613d4b408db7a4c7189ef7ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f73c2d3aac64266bf45eee00a10c095"
I20260927 15:10:01.368667 21948 raft_consensus.cc:3037] T 60b457e6e9fd4028afe96c14de957b33 P 8fac9d4b8f204372b80178dac5aa38cd [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:10:01.368685 21816 raft_consensus.cc:3037] T 60b457e6e9fd4028afe96c14de957b33 P 8f73c2d3aac64266bf45eee00a10c095 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:10:01.369992 21948 raft_consensus.cc:2454] T 60b457e6e9fd4028afe96c14de957b33 P 8fac9d4b8f204372b80178dac5aa38cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 881cc60f613d4b408db7a4c7189ef7ff in term 1.
I20260927 15:10:01.369990 21816 raft_consensus.cc:2454] T 60b457e6e9fd4028afe96c14de957b33 P 8f73c2d3aac64266bf45eee00a10c095 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 881cc60f613d4b408db7a4c7189ef7ff in term 1.
I20260927 15:10:01.370294 21612 leader_election.cc:302] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 881cc60f613d4b408db7a4c7189ef7ff, 8f73c2d3aac64266bf45eee00a10c095, 8fac9d4b8f204372b80178dac5aa38cd; no voters:
I20260927 15:10:01.370476 21730 raft_consensus.cc:2781] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:10:01.370687 21730 raft_consensus.cc:686] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 LEADER]: Becoming Leader. State: Replica: 881cc60f613d4b408db7a4c7189ef7ff, State: Running, Role: LEADER
I20260927 15:10:01.370951 21730 consensus_queue.cc:237] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } } peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } }
I20260927 15:10:01.373960 21536 catalog_manager.cc:5495] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 881cc60f613d4b408db7a4c7189ef7ff (127.7.17.193). New cstate: current_term: 1 leader_uuid: "881cc60f613d4b408db7a4c7189ef7ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } health_report { overall_health: HEALTHY } } }
I20260927 15:10:01.414036 22018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:01.410280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50266 (local address 127.7.17.193:43849)
0927 15:10:01.410600 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:01.410607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:01.410619 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:10:01.410695 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:01.410697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:01.410751 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:01.410859 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:10:01.411118 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.411660 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.412544 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:01.412776 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:01.412912 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:01.413178 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:10:01.413632 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:10:01.413737 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:01.413803 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 15:10:01.413863 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":127,"threads_started":1}
W20260927 15:10:01.416651 21995 tablet.cc:2369] T 60b457e6e9fd4028afe96c14de957b33 P 8fac9d4b8f204372b80178dac5aa38cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:10:01.418586 21948 raft_consensus.cc:1261] T 60b457e6e9fd4028afe96c14de957b33 P 8fac9d4b8f204372b80178dac5aa38cd [term 1 FOLLOWER]: Refusing update from remote peer 881cc60f613d4b408db7a4c7189ef7ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:10:01.418586 21816 raft_consensus.cc:1261] T 60b457e6e9fd4028afe96c14de957b33 P 8f73c2d3aac64266bf45eee00a10c095 [term 1 FOLLOWER]: Refusing update from remote peer 881cc60f613d4b408db7a4c7189ef7ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:10:01.419456 21730 consensus_queue.cc:1035] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:10:01.419691 22015 consensus_queue.cc:1035] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:10:01.427523 22019 mvcc.cc:205] Tried to move back new op lower bound from 7333977298602696704 to 7333977298418171904. Current Snapshot: MvccSnapshot[applied={T|T < 7333977298602696704}]
I20260927 15:10:01.439780 21531 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:33120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:10:01.439970 21531 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:33120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:10:01.442611 21531 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4: Sending DeleteTablet for 3 replicas of tablet 60b457e6e9fd4028afe96c14de957b33
I20260927 15:10:01.443342 21658 tablet_service.cc:1514] Processing DeleteTablet for tablet 60b457e6e9fd4028afe96c14de957b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:10:01 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:50264
I20260927 15:10:01.443445 21928 tablet_service.cc:1514] Processing DeleteTablet for tablet 60b457e6e9fd4028afe96c14de957b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:10:01 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:60598
I20260927 15:10:01.443451 21796 tablet_service.cc:1514] Processing DeleteTablet for tablet 60b457e6e9fd4028afe96c14de957b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:10:01 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:56066
I20260927 15:10:01.443984 22029 tablet_replica.cc:330] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff: stopping tablet replica
I20260927 15:10:01.444150 22029 raft_consensus.cc:2229] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:10:01.444146 22030 tablet_replica.cc:330] T 60b457e6e9fd4028afe96c14de957b33 P 8fac9d4b8f204372b80178dac5aa38cd: stopping tablet replica
I20260927 15:10:01.444214 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 21574
I20260927 15:10:01.445863 22030 raft_consensus.cc:2229] T 60b457e6e9fd4028afe96c14de957b33 P 8fac9d4b8f204372b80178dac5aa38cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:10:01.445960 22031 tablet_replica.cc:330] T 60b457e6e9fd4028afe96c14de957b33 P 8f73c2d3aac64266bf45eee00a10c095: stopping tablet replica
I20260927 15:10:01.446244 22030 raft_consensus.cc:2258] T 60b457e6e9fd4028afe96c14de957b33 P 8fac9d4b8f204372b80178dac5aa38cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:10:01.446270 22031 raft_consensus.cc:2229] T 60b457e6e9fd4028afe96c14de957b33 P 8f73c2d3aac64266bf45eee00a10c095 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:10:01.446563 22031 raft_consensus.cc:2258] T 60b457e6e9fd4028afe96c14de957b33 P 8f73c2d3aac64266bf45eee00a10c095 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:10:01.447732 22030 ts_tablet_manager.cc:1905] T 60b457e6e9fd4028afe96c14de957b33 P 8fac9d4b8f204372b80178dac5aa38cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:10:01.448024 22031 ts_tablet_manager.cc:1905] T 60b457e6e9fd4028afe96c14de957b33 P 8f73c2d3aac64266bf45eee00a10c095: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:10:01.450106 22030 ts_tablet_manager.cc:1918] T 60b457e6e9fd4028afe96c14de957b33 P 8fac9d4b8f204372b80178dac5aa38cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:10:01.450227 22030 log.cc:1196] T 60b457e6e9fd4028afe96c14de957b33 P 8fac9d4b8f204372b80178dac5aa38cd: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/60b457e6e9fd4028afe96c14de957b33
I20260927 15:10:01.450513 22030 ts_tablet_manager.cc:1939] T 60b457e6e9fd4028afe96c14de957b33 P 8fac9d4b8f204372b80178dac5aa38cd: Deleting consensus metadata
I20260927 15:10:01.450754 22031 ts_tablet_manager.cc:1918] T 60b457e6e9fd4028afe96c14de957b33 P 8f73c2d3aac64266bf45eee00a10c095: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:10:01.450896 22031 log.cc:1196] T 60b457e6e9fd4028afe96c14de957b33 P 8f73c2d3aac64266bf45eee00a10c095: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/wal/wals/60b457e6e9fd4028afe96c14de957b33
I20260927 15:10:01.451201 22031 ts_tablet_manager.cc:1939] T 60b457e6e9fd4028afe96c14de957b33 P 8f73c2d3aac64266bf45eee00a10c095: Deleting consensus metadata
I20260927 15:10:01.451241 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 21728
W20260927 15:10:01.452075 21521 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.17.193:43849 (error 108)
I20260927 15:10:01.452909 21522 catalog_manager.cc:4841] TS 8fac9d4b8f204372b80178dac5aa38cd (127.7.17.195:42987): tablet 60b457e6e9fd4028afe96c14de957b33 (table test-table [id=a4a97e060a7949bdba4642dfb076bf17]) successfully deleted
I20260927 15:10:01.453380 22004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:01.453099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:43849 (local address 127.0.0.1:50272)
0927 15:10:01.453229 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:01.453296 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.193:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 15:10:01.453599 21521 catalog_manager.cc:4584] TS 881cc60f613d4b408db7a4c7189ef7ff (127.7.17.193:43849): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 60b457e6e9fd4028afe96c14de957b33: Network error: Client connection negotiation failed: client connection to 127.7.17.193:43849: connect: Connection refused (error 111)
I20260927 15:10:01.456895 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 21865
I20260927 15:10:01.457726 22004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:01.457065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:42553 (local address 127.0.0.1:56082)
0927 15:10:01.457584 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:01.457639 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.194:42553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440}
I20260927 15:10:01.461512 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 21503
I20260927 15:10:01.467195 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:44411
--webserver_interface=127.7.17.254
--webserver_port=43913
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:44411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:10:01.578603 22033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:01.578904 22033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:01.578946 22033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:01.583169 22033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:10:01.583257 22033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:01.583276 22033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:01.583292 22033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:10:01.583305 22033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:10:01.589036 22033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:44411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:44411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=43913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:10:01.590523 22033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:01.591754 22033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:01.601339 22038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:01.601366 22041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:01.601336 22039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:01.601706 22033 server_base.cc:1034] running on GCE node
I20260927 15:10:01.602319 22033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:01.603636 22033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:01.604853 22033 hybrid_clock.cc:648] HybridClock initialized: now 1790521801604793 us; error 42 us; skew 500 ppm
Sep 27 15:10:01 dist-test-slave-hhpq krb5kdc[20955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521801, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:01.607577 22033 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:10:01.608487 22033 webserver.cc:460] Webserver started at http://127.7.17.254:43913/ using document root <none> and password file <none>
I20260927 15:10:01.609078 22033 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:01.609154 22033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:01.613001 22033 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:10:01.615149 22048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:01.616487 22033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:10:01.616607 22033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "f13f064244294db4b21bd6bf73cf2da4"
format_stamp: "Formatted at 2026-09-27 15:09:59 on dist-test-slave-hhpq"
server_key: "a92b9600e0e9d7a7d7b1c1183e037136"
server_key_iv: "c4743d41bcfb1755983e65f0d82596fb"
server_key_version: "encryptionkey@0"
I20260927 15:10:01.617213 22033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:01.631901 22033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:01.635270 22033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:01.635494 22033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:01.642903 22033 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:44411
I20260927 15:10:01.643170 22100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:44411 every 8 connection(s)
I20260927 15:10:01.644079 22033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:10:01.648643 22101 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:10:01.650755 22101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4: Bootstrap starting.
I20260927 15:10:01.653574 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 22033
I20260927 15:10:01.653997 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:43849
--local_ip_for_outbound_sockets=127.7.17.193
--tserver_master_addrs=127.7.17.254:44411
--webserver_port=34993
--webserver_interface=127.7.17.193
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:10:01.669055 22101 log.cc:824] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:01.681564 22101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:10:01.681957 22101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4: Bootstrap complete.
I20260927 15:10:01.686004 22101 raft_consensus.cc:348] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13f064244294db4b21bd6bf73cf2da4" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 44411 } }
I20260927 15:10:01.686432 22101 raft_consensus.cc:729] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f13f064244294db4b21bd6bf73cf2da4, State: Initialized, Role: FOLLOWER
I20260927 15:10:01.686928 22101 consensus_queue.cc:260] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13f064244294db4b21bd6bf73cf2da4" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 44411 } }
I20260927 15:10:01.687057 22101 raft_consensus.cc:388] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:10:01.687127 22101 raft_consensus.cc:482] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:10:01.687222 22101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 15:10:01.689307 22101 raft_consensus.cc:504] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13f064244294db4b21bd6bf73cf2da4" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 44411 } }
I20260927 15:10:01.689636 22101 leader_election.cc:302] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f13f064244294db4b21bd6bf73cf2da4; no voters:
I20260927 15:10:01.689955 22101 leader_election.cc:288] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 15:10:01.690059 22105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 15:10:01.690263 22105 raft_consensus.cc:686] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [term 3 LEADER]: Becoming Leader. State: Replica: f13f064244294db4b21bd6bf73cf2da4, State: Running, Role: LEADER
I20260927 15:10:01.690629 22105 consensus_queue.cc:237] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13f064244294db4b21bd6bf73cf2da4" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 44411 } }
I20260927 15:10:01.691256 22101 sys_catalog.cc:564] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:10:01.692117 22107 sys_catalog.cc:455] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f13f064244294db4b21bd6bf73cf2da4. Latest consensus state: current_term: 3 leader_uuid: "f13f064244294db4b21bd6bf73cf2da4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13f064244294db4b21bd6bf73cf2da4" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 44411 } } }
I20260927 15:10:01.692258 22107 sys_catalog.cc:458] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [sys.catalog]: This master's current role is: LEADER
I20260927 15:10:01.692533 22106 sys_catalog.cc:455] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f13f064244294db4b21bd6bf73cf2da4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13f064244294db4b21bd6bf73cf2da4" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 44411 } } }
I20260927 15:10:01.692610 22106 sys_catalog.cc:458] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4 [sys.catalog]: This master's current role is: LEADER
I20260927 15:10:01.696322 22120 catalog_manager.cc:1259] Loaded cluster ID: 313a2279e2cd4e638c59a691842e4f68
I20260927 15:10:01.696396 22120 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4: loading cluster ID for follower catalog manager: success
W20260927 15:10:01.697526 22120 catalog_manager.cc:1576] T 00000000000000000000000000000000 P f13f064244294db4b21bd6bf73cf2da4: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260927 15:10:01.697610 22120 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260927 15:10:01.697732 22114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:10:01.700661 22114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:10:01.701236 22114 catalog_manager.cc:1259] Loaded cluster ID: 313a2279e2cd4e638c59a691842e4f68
I20260927 15:10:01.701308 22114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 15:10:01.701964 22114 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f34f3c3e9ad google::LogMessage::Fail() at ??:0
@ 0x7f34f3c4188a google::LogMessage::SendToLog() at ??:0
@ 0x7f34f3c3e4de google::LogMessage::Flush() at ??:0
@ 0x7f34f3c3fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f34fe57dfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f34fe57ed2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f34fe57b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f34fe5bf640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f34f5e903d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f34f5e90c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f34f5e923f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f34f5e81512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f34f74c56db start_thread at ??:0
@ 0x7f34f273c61f clone at ??:0
W20260927 15:10:01.794423 22103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:01.794730 22103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:01.794790 22103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:01.799053 22103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:01.799140 22103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:01.799206 22103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:10:01.804859 22103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:43849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=34993
--tserver_master_addrs=127.7.17.254:44411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:10:01.806005 22103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:01.807073 22103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:01.817210 22143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:01.817178 22142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:01.817492 22145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:01.817395 22103 server_base.cc:1034] running on GCE node
I20260927 15:10:01.817919 22103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:01.818571 22103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:01.819727 22103 hybrid_clock.cc:648] HybridClock initialized: now 1790521801819703 us; error 45 us; skew 500 ppm
Sep 27 15:10:01 dist-test-slave-hhpq krb5kdc[20955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521801, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:01.822551 22103 init.cc:377] Logged in from keytab as kudu/127.7.17.193@KRBTEST.COM (short username kudu)
I20260927 15:10:01.823513 22103 webserver.cc:460] Webserver started at http://127.7.17.193:34993/ using document root <none> and password file <none>
I20260927 15:10:01.824070 22103 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:01.824136 22103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:01.828204 22103 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:10:01.830655 22152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:01.832304 22103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:10:01.832459 22103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "881cc60f613d4b408db7a4c7189ef7ff"
format_stamp: "Formatted at 2026-09-27 15:09:59 on dist-test-slave-hhpq"
server_key: "53a2aa6ca8989f8136f80a8d47d45b2d"
server_key_iv: "b024aea45856006327588994891f1540"
server_key_version: "encryptionkey@0"
I20260927 15:10:01.833247 22103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:01.854750 22103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:01.858103 22103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:01.858295 22103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:01.858886 22103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:01.860005 22159 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 15:10:01.865558 22103 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 15:10:01.865645 22103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 15:10:01.865726 22103 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 15:10:01.867461 22103 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 15:10:01.867511 22103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 15:10:01.867898 22159 tablet_bootstrap.cc:492] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff: Bootstrap starting.
I20260927 15:10:01.877526 22103 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:43849
I20260927 15:10:01.877735 22266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:43849 every 8 connection(s)
I20260927 15:10:01.878700 22103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:10:01.881354 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 22103
I20260927 15:10:01.881821 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:42553
--local_ip_for_outbound_sockets=127.7.17.194
--tserver_master_addrs=127.7.17.254:44411
--webserver_port=40863
--webserver_interface=127.7.17.194
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:10:01.890046 22159 log.cc:824] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:01.899636 22159 tablet_bootstrap.cc:492] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:10:01.899998 22159 tablet_bootstrap.cc:492] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff: Bootstrap complete.
I20260927 15:10:01.901347 22159 ts_tablet_manager.cc:1397] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff: Time spent bootstrapping tablet: real 0.034s user 0.025s sys 0.008s
I20260927 15:10:01.904855 22159 raft_consensus.cc:348] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } } peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } }
I20260927 15:10:01.905440 22159 raft_consensus.cc:729] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881cc60f613d4b408db7a4c7189ef7ff, State: Initialized, Role: FOLLOWER
I20260927 15:10:01.906028 22159 consensus_queue.cc:260] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } } peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } }
I20260927 15:10:01.907169 22159 ts_tablet_manager.cc:1428] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260927 15:10:01.968144 22269 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:10:01.881110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.193:48981)
0927 15:10:01.882028 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:01.882087 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:01.883117 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:01.967285 (+ 84168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":177,"threads_started":1}
W20260927 15:10:01.968645 22267 heartbeater.cc:644] Failed to heartbeat to 127.7.17.254:44411 (0 consecutive failures): Network error: Failed to ping master at 127.7.17.254:44411: Client connection negotiation failed: client connection to 127.7.17.254:44411: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 15:10:01.969463 22269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:01.969163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.193:56977)
0927 15:10:01.969289 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:01.969372 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 15:10:01.970156 22269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:01.969914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.193:45953)
0927 15:10:01.970020 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:01.970097 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 15:10:01.970309 22267 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:10:01.998402 22270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:01.998688 22270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:01.998759 22270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:02.002904 22270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:02.002985 22270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:02.003068 22270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:10:02.008528 22270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:42553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=40863
--tserver_master_addrs=127.7.17.254:44411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:10:02.009657 22270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:02.010686 22270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:02.020457 22280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:02.020486 22277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:02.020486 22278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:02.020609 22270 server_base.cc:1034] running on GCE node
I20260927 15:10:02.021096 22270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:02.021766 22270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:02.022943 22270 hybrid_clock.cc:648] HybridClock initialized: now 1790521802022917 us; error 38 us; skew 500 ppm
Sep 27 15:10:02 dist-test-slave-hhpq krb5kdc[20955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521802, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:02.025591 22270 init.cc:377] Logged in from keytab as kudu/127.7.17.194@KRBTEST.COM (short username kudu)
I20260927 15:10:02.026450 22270 webserver.cc:460] Webserver started at http://127.7.17.194:40863/ using document root <none> and password file <none>
I20260927 15:10:02.026969 22270 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:02.027035 22270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:02.031107 22270 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:10:02.033365 22287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:02.034796 22270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:10:02.034943 22270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "8f73c2d3aac64266bf45eee00a10c095"
format_stamp: "Formatted at 2026-09-27 15:09:59 on dist-test-slave-hhpq"
server_key: "961d8a9a99c24315704f0c53399c3f39"
server_key_iv: "2937772d44511fc1862e8fc36057298a"
server_key_version: "encryptionkey@0"
I20260927 15:10:02.035547 22270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:02.080154 22270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:02.083874 22270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:02.084090 22270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:02.084746 22270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:02.085680 22270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:10:02.085731 22270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:02.085788 22270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:10:02.085813 22270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:02.095105 22270 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:42553
I20260927 15:10:02.095130 22400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:42553 every 8 connection(s)
I20260927 15:10:02.096184 22270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:10:02.099442 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 22270
I20260927 15:10:02.099861 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:42987
--local_ip_for_outbound_sockets=127.7.17.195
--tserver_master_addrs=127.7.17.254:44411
--webserver_port=46373
--webserver_interface=127.7.17.195
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:10:02.100441 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.098524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.194:52081)
0927 15:10:02.099014 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:02.099544 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":115,"threads_started":1}
W20260927 15:10:02.100982 22401 heartbeater.cc:644] Failed to heartbeat to 127.7.17.254:44411 (0 consecutive failures): Network error: Failed to ping master at 127.7.17.254:44411: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
I20260927 15:10:02.101809 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.101496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.194:33529)
0927 15:10:02.101643 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:02.101750 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 15:10:02.102437 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.102243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.194:38923)
0927 15:10:02.102342 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:02.102388 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 15:10:02.102584 22401 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:10:02.213073 22404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:02.213383 22404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:02.213442 22404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:02.217758 22404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:02.217833 22404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:02.217922 22404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:10:02.223397 22404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:42987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=46373
--tserver_master_addrs=127.7.17.254:44411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:10:02.224548 22404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:02.225630 22404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:02.235378 22410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:02.235357 22409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:02.235361 22412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:02.235721 22404 server_base.cc:1034] running on GCE node
I20260927 15:10:02.236132 22404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:02.236773 22404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:02.237954 22404 hybrid_clock.cc:648] HybridClock initialized: now 1790521802237935 us; error 33 us; skew 500 ppm
Sep 27 15:10:02 dist-test-slave-hhpq krb5kdc[20955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521802, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:02.240609 22404 init.cc:377] Logged in from keytab as kudu/127.7.17.195@KRBTEST.COM (short username kudu)
I20260927 15:10:02.241531 22404 webserver.cc:460] Webserver started at http://127.7.17.195:46373/ using document root <none> and password file <none>
I20260927 15:10:02.242079 22404 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:02.242142 22404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:02.246060 22404 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 15:10:02.248104 22419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:02.249454 22404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:10:02.249554 22404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "8fac9d4b8f204372b80178dac5aa38cd"
format_stamp: "Formatted at 2026-09-27 15:09:59 on dist-test-slave-hhpq"
server_key: "bf99e66287c79017e81df9bc48ab2d7c"
server_key_iv: "9201de33ac3b0045d787800f99775c6f"
server_key_version: "encryptionkey@0"
I20260927 15:10:02.250144 22404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:02.265707 22404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:02.269008 22404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:02.269191 22404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:02.269796 22404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:02.270691 22404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:10:02.270738 22404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:02.270792 22404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:10:02.270818 22404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:02.279654 22404 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:42987
I20260927 15:10:02.279687 22532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:42987 every 8 connection(s)
I20260927 15:10:02.280716 22404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:10:02.284977 22535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.283109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.195:42679)
0927 15:10:02.283660 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:02.284164 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":132,"threads_started":1}
W20260927 15:10:02.285548 22533 heartbeater.cc:644] Failed to heartbeat to 127.7.17.254:44411 (0 consecutive failures): Network error: Failed to ping master at 127.7.17.254:44411: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
I20260927 15:10:02.286369 22535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.286109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.195:50669)
0927 15:10:02.286233 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:02.286286 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 15:10:02.286648 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 22404
I20260927 15:10:02.287094 22535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.286832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.195:55279)
0927 15:10:02.286991 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:02.287038 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260927 15:10:02.287297 22533 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 15:10:02.971493 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:02.971064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.193:54177)
0927 15:10:02.971317 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:02.971377 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260927 15:10:03.103657 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:03.103206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.194:52857)
0927 15:10:03.103457 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:03.103519 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":72,"threads_started":1}
I20260927 15:10:03.189026 22539 raft_consensus.cc:482] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:10:03.189234 22539 raft_consensus.cc:504] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } } peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } }
I20260927 15:10:03.190429 22539 leader_election.cc:288] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f73c2d3aac64266bf45eee00a10c095 (127.7.17.194:42553), 8fac9d4b8f204372b80178dac5aa38cd (127.7.17.195:42987)
Sep 27 15:10:03 dist-test-slave-hhpq krb5kdc[20955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521801, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for kudu/127.7.17.194@KRBTEST.COM
Sep 27 15:10:03 dist-test-slave-hhpq krb5kdc[20955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521801, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for kudu/127.7.17.195@KRBTEST.COM
I20260927 15:10:03.204850 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:03.190685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:42553 (local address 127.7.17.193:42349)
0927 15:10:03.191054 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:03.191064 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:03.191134 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:03.192932 (+ 1798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:03.192938 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:03.193497 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:10:03.194013 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:03.194030 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:03.195309 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:03.195311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:03.196284 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:03.196294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:03.197061 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:03.197065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:03.197279 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:03.197958 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:10:03.197970 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:10:03.199539 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:10:03.202026 (+ 2487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:03.202033 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:03.202051 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:03.202533 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:03.202950 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:03.202954 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:03.202956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:03.203152 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:03.203842 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:10:03.203848 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:10:03.204093 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 15:10:03.204274 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 15:10:03.204628 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":213,"spinlock_wait_cycles":31360,"thread_start_us":82,"threads_started":1}
I20260927 15:10:03.204867 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:03.190863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:57425 (local address 127.7.17.195:42987)
0927 15:10:03.191341 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:03.191356 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:03.191389 (+ 33us) server_negotiation.cc:407] Connection header received
0927 15:10:03.192148 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:03.192157 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:03.192509 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:03.192804 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:03.194193 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:03.195161 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:03.196411 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:03.196942 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:03.197093 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:03.200129 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:03.200137 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:03.200155 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:03.200192 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:03.201884 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:03.202732 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:03.202739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:03.202744 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:03.202871 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:03.203239 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:03.203243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:03.203245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:03.203723 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:03.203845 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:03.204403 (+ 558us) server_negotiation.cc:299] Negotiation successful
0927 15:10:03.204667 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":108,"threads_started":1}
I20260927 15:10:03.204895 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:03.190862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:42349 (local address 127.7.17.194:42553)
0927 15:10:03.191247 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:03.191268 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:03.191308 (+ 40us) server_negotiation.cc:407] Connection header received
0927 15:10:03.192088 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:03.192098 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:03.192510 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:03.192804 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:03.194194 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:03.195158 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:03.196454 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:03.196941 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:03.197100 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:03.199807 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:03.199813 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:03.199829 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:03.199863 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:03.201884 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:03.202676 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:03.202683 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:03.202690 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:03.202854 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:03.203255 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:03.203258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:03.203259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:03.203716 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:03.203834 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:03.204418 (+ 584us) server_negotiation.cc:299] Negotiation successful
0927 15:10:03.204677 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":65,"threads_started":1}
I20260927 15:10:03.204883 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:03.190685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:42987 (local address 127.7.17.193:57425)
0927 15:10:03.190961 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:03.190979 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:03.191094 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:03.192931 (+ 1837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:03.192938 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:03.193490 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:10:03.194013 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:03.194030 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:03.195300 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:03.195303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:03.196285 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:03.196296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:03.197061 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:03.197074 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:03.197283 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:03.197923 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:10:03.197949 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:10:03.199918 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:10:03.202026 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:03.202033 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:03.202050 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:03.202533 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:03.202976 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:03.202979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:03.202981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:03.203140 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:03.203841 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:10:03.203848 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:10:03.204095 (+ 247us) client_negotiation.cc:769] Sending connection context
0927 15:10:03.204256 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 15:10:03.204713 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":349,"spinlock_wait_cycles":65024}
W20260927 15:10:03.207820 22156 leader_election.cc:341] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f73c2d3aac64266bf45eee00a10c095 (127.7.17.194:42553): Not found: Tablet not found: 60b457e6e9fd4028afe96c14de957b33
W20260927 15:10:03.208534 22155 leader_election.cc:341] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8fac9d4b8f204372b80178dac5aa38cd (127.7.17.195:42987): Not found: Tablet not found: 60b457e6e9fd4028afe96c14de957b33
I20260927 15:10:03.208621 22155 leader_election.cc:302] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 881cc60f613d4b408db7a4c7189ef7ff; no voters: 8f73c2d3aac64266bf45eee00a10c095, 8fac9d4b8f204372b80178dac5aa38cd
I20260927 15:10:03.208889 22539 raft_consensus.cc:2726] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:10:03.288436 22543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:03.287991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.195:49379)
0927 15:10:03.288267 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:03.288331 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":74,"threads_started":1}
I20260927 15:10:03.972779 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:03.972339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.193:47547)
0927 15:10:03.972615 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:03.972682 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":81,"threads_started":1}
I20260927 15:10:04.105048 22545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:04.104523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.194:44895)
0927 15:10:04.104870 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:04.104945 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":78,"threads_started":1}
I20260927 15:10:04.289861 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:04.289344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.195:55019)
0927 15:10:04.289635 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:04.289702 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":80,"threads_started":1}
I20260927 15:10:04.771178 22547 raft_consensus.cc:482] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:10:04.771308 22547 raft_consensus.cc:504] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } } peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } }
I20260927 15:10:04.771728 22547 leader_election.cc:288] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f73c2d3aac64266bf45eee00a10c095 (127.7.17.194:42553), 8fac9d4b8f204372b80178dac5aa38cd (127.7.17.195:42987)
W20260927 15:10:04.772583 22156 leader_election.cc:341] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f73c2d3aac64266bf45eee00a10c095 (127.7.17.194:42553): Not found: Tablet not found: 60b457e6e9fd4028afe96c14de957b33
W20260927 15:10:04.772717 22155 leader_election.cc:341] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8fac9d4b8f204372b80178dac5aa38cd (127.7.17.195:42987): Not found: Tablet not found: 60b457e6e9fd4028afe96c14de957b33
I20260927 15:10:04.772786 22155 leader_election.cc:302] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 881cc60f613d4b408db7a4c7189ef7ff; no voters: 8f73c2d3aac64266bf45eee00a10c095, 8fac9d4b8f204372b80178dac5aa38cd
I20260927 15:10:04.772953 22547 raft_consensus.cc:2726] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:10:04.974038 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:04.973567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.193:36531)
0927 15:10:04.973879 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:04.973940 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":151,"threads_started":1}
I20260927 15:10:05.106292 22549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:05.105851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.194:55977)
0927 15:10:05.106113 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:05.106179 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260927 15:10:05.291131 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:05.290638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.195:50423)
0927 15:10:05.290936 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:05.291005 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":79,"threads_started":1}
I20260927 15:10:05.975251 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:05.974823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.193:52259)
0927 15:10:05.975093 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:05.975155 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":79,"threads_started":1}
I20260927 15:10:06.107570 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:06.107123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.194:56767)
0927 15:10:06.107406 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:06.107472 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":80,"threads_started":1}
I20260927 15:10:06.292630 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:06.292107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.195:36407)
0927 15:10:06.292461 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:06.292529 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":90,"threads_started":1}
I20260927 15:10:06.384024 22554 raft_consensus.cc:482] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:10:06.384169 22554 raft_consensus.cc:504] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } } peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } }
I20260927 15:10:06.384542 22554 leader_election.cc:288] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f73c2d3aac64266bf45eee00a10c095 (127.7.17.194:42553), 8fac9d4b8f204372b80178dac5aa38cd (127.7.17.195:42987)
W20260927 15:10:06.385357 22156 leader_election.cc:341] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f73c2d3aac64266bf45eee00a10c095 (127.7.17.194:42553): Not found: Tablet not found: 60b457e6e9fd4028afe96c14de957b33
W20260927 15:10:06.385473 22155 leader_election.cc:341] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8fac9d4b8f204372b80178dac5aa38cd (127.7.17.195:42987): Not found: Tablet not found: 60b457e6e9fd4028afe96c14de957b33
I20260927 15:10:06.385557 22155 leader_election.cc:302] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 881cc60f613d4b408db7a4c7189ef7ff; no voters: 8f73c2d3aac64266bf45eee00a10c095, 8fac9d4b8f204372b80178dac5aa38cd
I20260927 15:10:06.385744 22554 raft_consensus.cc:2726] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:10:06.976537 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:06.976091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.193:56097)
0927 15:10:06.976393 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:06.976445 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":72,"threads_started":1}
I20260927 15:10:07.108807 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:07.108404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.194:37489)
0927 15:10:07.108656 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:07.108711 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260927 15:10:07.294039 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:07.293591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.195:33343)
0927 15:10:07.293869 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:07.293939 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":80,"threads_started":1}
I20260927 15:10:07.977862 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:07.977374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.193:34415)
0927 15:10:07.977691 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:07.977752 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":108,"threads_started":1}
I20260927 15:10:08.110162 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:08.109686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.194:36347)
0927 15:10:08.110008 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:08.110064 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":93,"threads_started":1}
I20260927 15:10:08.295441 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:08.294951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.195:46409)
0927 15:10:08.295272 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:08.295341 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":102,"threads_started":1}
I20260927 15:10:08.979111 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:08.978642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.193:52963)
0927 15:10:08.978948 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:08.979013 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":93,"threads_started":1}
I20260927 15:10:09.111431 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:09.110951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.194:52341)
0927 15:10:09.111267 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:09.111331 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":75,"threads_started":1}
I20260927 15:10:09.296934 22563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:09.296332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.195:50575)
0927 15:10:09.296698 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:09.296775 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":83,"threads_started":1}
I20260927 15:10:09.980346 22564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:09.979925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.193:34075)
0927 15:10:09.980181 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:09.980245 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
I20260927 15:10:10.077368 22565 raft_consensus.cc:482] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:10:10.077508 22565 raft_consensus.cc:504] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } } peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } }
I20260927 15:10:10.077914 22565 leader_election.cc:288] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f73c2d3aac64266bf45eee00a10c095 (127.7.17.194:42553), 8fac9d4b8f204372b80178dac5aa38cd (127.7.17.195:42987)
W20260927 15:10:10.078759 22155 leader_election.cc:341] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8fac9d4b8f204372b80178dac5aa38cd (127.7.17.195:42987): Not found: Tablet not found: 60b457e6e9fd4028afe96c14de957b33
W20260927 15:10:10.078883 22156 leader_election.cc:341] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f73c2d3aac64266bf45eee00a10c095 (127.7.17.194:42553): Not found: Tablet not found: 60b457e6e9fd4028afe96c14de957b33
I20260927 15:10:10.079061 22156 leader_election.cc:302] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 881cc60f613d4b408db7a4c7189ef7ff; no voters: 8f73c2d3aac64266bf45eee00a10c095, 8fac9d4b8f204372b80178dac5aa38cd
I20260927 15:10:10.079221 22565 raft_consensus.cc:2726] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:10:10.112715 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:10.112251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.194:43987)
0927 15:10:10.112554 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:10.112619 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":87,"threads_started":1}
I20260927 15:10:10.298269 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:10.297825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.195:49427)
0927 15:10:10.298102 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:10.298167 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":81,"threads_started":1}
I20260927 15:10:10.981666 22568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:10.981207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.193:54309)
0927 15:10:10.981500 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:10.981565 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":95,"threads_started":1}
I20260927 15:10:11.114013 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:11.113564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.194:56653)
0927 15:10:11.113853 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:11.113917 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":88,"threads_started":1}
I20260927 15:10:11.299584 22570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:11.299123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.195:45409)
0927 15:10:11.299411 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:11.299483 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":87,"threads_started":1}
I20260927 15:10:11.983048 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:11.982606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.193:48311)
0927 15:10:11.982881 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:11.982948 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
I20260927 15:10:12.115320 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:12.114864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.194:44571)
0927 15:10:12.115131 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:12.115205 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":80,"threads_started":1}
I20260927 15:10:12.301040 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:12.300530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.195:50727)
0927 15:10:12.300863 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:12.300940 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":81,"threads_started":1}
I20260927 15:10:12.721308 22574 raft_consensus.cc:482] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:10:12.721463 22574 raft_consensus.cc:504] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } } peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } }
I20260927 15:10:12.721899 22574 leader_election.cc:288] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f73c2d3aac64266bf45eee00a10c095 (127.7.17.194:42553), 8fac9d4b8f204372b80178dac5aa38cd (127.7.17.195:42987)
W20260927 15:10:12.722693 22156 leader_election.cc:341] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f73c2d3aac64266bf45eee00a10c095 (127.7.17.194:42553): Not found: Tablet not found: 60b457e6e9fd4028afe96c14de957b33
W20260927 15:10:12.722815 22155 leader_election.cc:341] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8fac9d4b8f204372b80178dac5aa38cd (127.7.17.195:42987): Not found: Tablet not found: 60b457e6e9fd4028afe96c14de957b33
I20260927 15:10:12.722895 22155 leader_election.cc:302] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 881cc60f613d4b408db7a4c7189ef7ff; no voters: 8f73c2d3aac64266bf45eee00a10c095, 8fac9d4b8f204372b80178dac5aa38cd
I20260927 15:10:12.723047 22574 raft_consensus.cc:2726] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:10:12.984450 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:12.983979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.193:48743)
0927 15:10:12.984285 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:12.984354 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":95,"threads_started":1}
I20260927 15:10:13.116515 22576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:13.116094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.194:41459)
0927 15:10:13.116357 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:13.116418 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":79,"threads_started":1}
I20260927 15:10:13.302299 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:13.301828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.195:55145)
0927 15:10:13.302124 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:13.302192 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":97,"threads_started":1}
I20260927 15:10:13.985841 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:13.985390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.193:45667)
0927 15:10:13.985664 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:13.985740 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":78,"threads_started":1}
I20260927 15:10:14.117786 22579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:14.117344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.194:47103)
0927 15:10:14.117619 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:14.117687 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":76,"threads_started":1}
I20260927 15:10:14.303565 22580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:14.303151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.195:35897)
0927 15:10:14.303405 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:14.303467 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":100,"threads_started":1}
I20260927 15:10:14.987160 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:14.986649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.193:34503)
0927 15:10:14.986954 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:14.987021 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":87,"threads_started":1}
I20260927 15:10:15.118983 22582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:15.118575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.194:40711)
0927 15:10:15.118824 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:15.118885 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":88,"threads_started":1}
I20260927 15:10:15.296581 22583 raft_consensus.cc:482] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:10:15.296746 22583 raft_consensus.cc:504] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73c2d3aac64266bf45eee00a10c095" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42553 } } peers { permanent_uuid: "8fac9d4b8f204372b80178dac5aa38cd" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 42987 } } peers { permanent_uuid: "881cc60f613d4b408db7a4c7189ef7ff" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 43849 } }
I20260927 15:10:15.297226 22583 leader_election.cc:288] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f73c2d3aac64266bf45eee00a10c095 (127.7.17.194:42553), 8fac9d4b8f204372b80178dac5aa38cd (127.7.17.195:42987)
W20260927 15:10:15.298100 22155 leader_election.cc:341] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8fac9d4b8f204372b80178dac5aa38cd (127.7.17.195:42987): Not found: Tablet not found: 60b457e6e9fd4028afe96c14de957b33
W20260927 15:10:15.298234 22156 leader_election.cc:341] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f73c2d3aac64266bf45eee00a10c095 (127.7.17.194:42553): Not found: Tablet not found: 60b457e6e9fd4028afe96c14de957b33
I20260927 15:10:15.298317 22156 leader_election.cc:302] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 881cc60f613d4b408db7a4c7189ef7ff; no voters: 8f73c2d3aac64266bf45eee00a10c095, 8fac9d4b8f204372b80178dac5aa38cd
I20260927 15:10:15.298452 22583 raft_consensus.cc:2726] T 60b457e6e9fd4028afe96c14de957b33 P 881cc60f613d4b408db7a4c7189ef7ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:10:15.304944 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:15.304430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.195:59085)
0927 15:10:15.304706 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:15.304777 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":90,"threads_started":1}
I20260927 15:10:15.988466 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:15.988017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.193:56537)
0927 15:10:15.988306 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:15.988368 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":106,"threads_started":1}
I20260927 15:10:16.120287 22586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:16.119828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.194:39537)
0927 15:10:16.120112 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:16.120182 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":76,"threads_started":1}
I20260927 15:10:16.306277 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:16.305800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.195:49313)
0927 15:10:16.306113 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:16.306180 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":104,"threads_started":1}
I20260927 15:10:16.989946 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:16.989513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.193:56941)
0927 15:10:16.989786 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:16.989849 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":73,"threads_started":1}
I20260927 15:10:17.121685 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:17.121229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:44411 (local address 127.7.17.194:40529)
0927 15:10:17.121518 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:17.121585 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":86,"threads_started":1}
I20260927 15:10:17.292254 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 22103
I20260927 15:10:17.297503 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 22270
I20260927 15:10:17.302631 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 22404
2026-09-27T15:10:17Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19274 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:10:17 dist-test-slave-hhpq krb5kdc[22592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:10:17 dist-test-slave-hhpq krb5kdc[22592](info): set up 2 sockets
Sep 27 15:10:17 dist-test-slave-hhpq krb5kdc[22592](info): commencing operation
krb5kdc: starting...
W20260927 15:10:18.265002 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:10:18 dist-test-slave-hhpq krb5kdc[22592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:10:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:10:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:10:18.387825 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:46765
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:46765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:10:18.503563 22605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:18.503865 22605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:18.503904 22605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:18.508080 22605 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 15:10:18.508160 22605 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 15:10:18.508175 22605 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 15:10:18.508188 22605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:10:18.508205 22605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:18.508221 22605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:18.508237 22605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:10:18.508257 22605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:10:18.513851 22605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:46765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:46765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:10:18.514982 22605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:18.516059 22605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:10:18.525480 22605 server_base.cc:1034] running on GCE node
W20260927 15:10:18.525650 22613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:18.525514 22610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:18.525538 22611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:18.526471 22605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:18.527683 22605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:18.528891 22605 hybrid_clock.cc:648] HybridClock initialized: now 1790521818528873 us; error 31 us; skew 500 ppm
Sep 27 15:10:18 dist-test-slave-hhpq krb5kdc[22592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521818, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:18.531708 22605 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:10:18.532670 22605 webserver.cc:460] Webserver started at http://127.7.17.254:36883/ using document root <none> and password file <none>
I20260927 15:10:18.533279 22605 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:18.533350 22605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:18.533587 22605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:10:18.535812 22605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "af580a3e658c4e8a9731da421183718b"
format_stamp: "Formatted at 2026-09-27 15:10:18 on dist-test-slave-hhpq"
server_key: "1364626bf50f3700bf560ab9017792db"
server_key_iv: "b4bda720f322c0eb339345d71e7d85fe"
server_key_version: "encryptionkey@0"
I20260927 15:10:18.536407 22605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "af580a3e658c4e8a9731da421183718b"
format_stamp: "Formatted at 2026-09-27 15:10:18 on dist-test-slave-hhpq"
server_key: "1364626bf50f3700bf560ab9017792db"
server_key_iv: "b4bda720f322c0eb339345d71e7d85fe"
server_key_version: "encryptionkey@0"
I20260927 15:10:18.540472 22605 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:10:18.543211 22620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:18.544870 22605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:10:18.545015 22605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "af580a3e658c4e8a9731da421183718b"
format_stamp: "Formatted at 2026-09-27 15:10:18 on dist-test-slave-hhpq"
server_key: "1364626bf50f3700bf560ab9017792db"
server_key_iv: "b4bda720f322c0eb339345d71e7d85fe"
server_key_version: "encryptionkey@0"
I20260927 15:10:18.545128 22605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:18.565799 22605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:18.569192 22605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:18.569380 22605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:18.576457 22605 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:46765
I20260927 15:10:18.576750 22672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:46765 every 8 connection(s)
I20260927 15:10:18.577518 22605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:10:18.580569 22673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:10:18.584458 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 22605
I20260927 15:10:18.584586 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:10:18.584993 7239 external_mini_cluster.cc:1428] Setting key 394e4841df251d2a957c20932b5db8f1
I20260927 15:10:18.587239 22673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b: Bootstrap starting.
I20260927 15:10:18.589541 22673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b: Neither blocks nor log segments found. Creating new log.
I20260927 15:10:18.590252 22673 log.cc:824] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:18.592257 22673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b: No bootstrap required, opened a new log
Sep 27 15:10:18 dist-test-slave-hhpq krb5kdc[22592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:10:18.595839 22673 raft_consensus.cc:348] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af580a3e658c4e8a9731da421183718b" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46765 } }
I20260927 15:10:18.596081 22673 raft_consensus.cc:374] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:10:18.596148 22673 raft_consensus.cc:729] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af580a3e658c4e8a9731da421183718b, State: Initialized, Role: FOLLOWER
I20260927 15:10:18.596654 22673 consensus_queue.cc:260] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af580a3e658c4e8a9731da421183718b" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46765 } }
I20260927 15:10:18.596817 22673 raft_consensus.cc:388] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:10:18.596928 22673 raft_consensus.cc:482] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:10:18.597036 22673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:10:18.598223 22673 raft_consensus.cc:504] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af580a3e658c4e8a9731da421183718b" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46765 } }
I20260927 15:10:18.598605 22673 leader_election.cc:302] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af580a3e658c4e8a9731da421183718b; no voters:
I20260927 15:10:18.598934 22673 leader_election.cc:288] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:10:18.599149 22678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:10:18.599383 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:18.586581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37406 (local address 127.7.17.254:46765)
0927 15:10:18.587044 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:18.587060 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:18.587097 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:10:18.587844 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:18.587865 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:18.588228 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:18.588545 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:18.589356 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:18.590385 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:18.591385 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:18.591861 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:18.592012 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:18.594357 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:18.594363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:18.594377 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:18.594405 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:18.595974 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:18.596458 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:18.596464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:18.596470 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:18.596616 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:18.596913 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:18.596917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:18.596919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:18.597463 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:18.597662 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:18.597980 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 15:10:18.598337 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":145,"threads_started":1}
I20260927 15:10:18.599397 22678 raft_consensus.cc:686] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 1 LEADER]: Becoming Leader. State: Replica: af580a3e658c4e8a9731da421183718b, State: Running, Role: LEADER
I20260927 15:10:18.599979 22678 consensus_queue.cc:237] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af580a3e658c4e8a9731da421183718b" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46765 } }
I20260927 15:10:18.600286 22673 sys_catalog.cc:564] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:10:18.601747 22680 sys_catalog.cc:455] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [sys.catalog]: SysCatalogTable state changed. Reason: New leader af580a3e658c4e8a9731da421183718b. Latest consensus state: current_term: 1 leader_uuid: "af580a3e658c4e8a9731da421183718b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af580a3e658c4e8a9731da421183718b" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46765 } } }
I20260927 15:10:18.601778 22679 sys_catalog.cc:455] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af580a3e658c4e8a9731da421183718b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af580a3e658c4e8a9731da421183718b" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46765 } } }
I20260927 15:10:18.602003 22679 sys_catalog.cc:458] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [sys.catalog]: This master's current role is: LEADER
I20260927 15:10:18.602262 22680 sys_catalog.cc:458] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [sys.catalog]: This master's current role is: LEADER
I20260927 15:10:18.602403 22691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:10:18.606070 22691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:10:18.612267 22691 catalog_manager.cc:1347] Generated new cluster ID: 936dd6b417434a0c971f8674f81c4d4a
I20260927 15:10:18.612361 22691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:10:18.639503 22691 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:10:18.640925 22691 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:10:18.650753 22691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b: Generated new TSK 0
I20260927 15:10:18.651674 22691 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
I20260927 15:10:18.692528 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46765
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:10:18.806763 22699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:18.807057 22699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:18.807106 22699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:18.811321 22699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:18.811403 22699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:18.811468 22699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:10:18.817101 22699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:10:18.818228 22699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:18.819242 22699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:18.828707 22707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:18.828863 22704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:18.828699 22705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:18.829114 22699 server_base.cc:1034] running on GCE node
I20260927 15:10:18.829504 22699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:18.830160 22699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:18.831326 22699 hybrid_clock.cc:648] HybridClock initialized: now 1790521818831298 us; error 40 us; skew 500 ppm
Sep 27 15:10:18 dist-test-slave-hhpq krb5kdc[22592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521818, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:18.834141 22699 init.cc:377] Logged in from keytab as kudu/127.7.17.193@KRBTEST.COM (short username kudu)
I20260927 15:10:18.835041 22699 webserver.cc:460] Webserver started at http://127.7.17.193:36101/ using document root <none> and password file <none>
I20260927 15:10:18.835618 22699 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:18.835683 22699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:18.835878 22699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:10:18.838160 22699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "0a1b0c9c4d49448f9d40a69effa00d21"
format_stamp: "Formatted at 2026-09-27 15:10:18 on dist-test-slave-hhpq"
server_key: "47927d8866c8b84da96eec2b19045bf3"
server_key_iv: "d7b72d30db6d4f10bde001e5f95ae503"
server_key_version: "encryptionkey@0"
I20260927 15:10:18.838726 22699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "0a1b0c9c4d49448f9d40a69effa00d21"
format_stamp: "Formatted at 2026-09-27 15:10:18 on dist-test-slave-hhpq"
server_key: "47927d8866c8b84da96eec2b19045bf3"
server_key_iv: "d7b72d30db6d4f10bde001e5f95ae503"
server_key_version: "encryptionkey@0"
I20260927 15:10:18.842448 22699 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:10:18.845021 22714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:18.846268 22699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:10:18.846374 22699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "0a1b0c9c4d49448f9d40a69effa00d21"
format_stamp: "Formatted at 2026-09-27 15:10:18 on dist-test-slave-hhpq"
server_key: "47927d8866c8b84da96eec2b19045bf3"
server_key_iv: "d7b72d30db6d4f10bde001e5f95ae503"
server_key_version: "encryptionkey@0"
I20260927 15:10:18.846472 22699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:18.867604 22699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:18.871131 22699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:18.871322 22699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:18.871948 22699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:18.872893 22699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:10:18.872944 22699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:18.873003 22699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:10:18.873030 22699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:18.881774 22699 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:32919
I20260927 15:10:18.881801 22827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:32919 every 8 connection(s)
I20260927 15:10:18.882752 22699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:10:18.889492 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 22699
I20260927 15:10:18.889616 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:10:18.889889 7239 external_mini_cluster.cc:1428] Setting key 6db857a24ce292678344c601332e71d9
Sep 27 15:10:18 dist-test-slave-hhpq krb5kdc[22592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521818, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:10:18.897750 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:18.884698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:52571 (local address 127.7.17.254:46765)
0927 15:10:18.884875 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:18.884880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:18.886019 (+ 1139us) server_negotiation.cc:407] Connection header received
0927 15:10:18.887340 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:18.887343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:18.887404 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:18.887492 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:18.888758 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:18.889363 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:18.890761 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:18.891009 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:18.891104 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:18.893984 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:18.893987 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:18.893989 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:18.894015 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:18.895559 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:18.896093 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:18.896097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:18.896099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:18.896195 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:18.896502 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:18.896505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:18.896507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:18.896759 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:18.896887 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:18.897426 (+ 539us) server_negotiation.cc:299] Negotiation successful
0927 15:10:18.897564 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:10:18.898550 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:18.885290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.193:52571)
0927 15:10:18.885843 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:18.885896 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:18.887105 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:18.887707 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:18.887715 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:18.888100 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:10:18.888600 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:18.888614 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:18.889489 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:18.889492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:18.890604 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:18.890617 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:18.891203 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:18.891208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:18.891415 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:18.892102 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:10:18.892120 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:10:18.893785 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:10:18.895693 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:18.895699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:18.895715 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:18.895989 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:18.896280 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:18.896282 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:18.896283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:18.896420 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:18.896881 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:10:18.896888 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:10:18.897187 (+ 299us) client_negotiation.cc:769] Sending connection context
0927 15:10:18.897356 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 15:10:18.897628 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":124,"threads_started":1}
WARNING: no policy specified for kudu/127.7.17.194@KRBTEST.COM; defaulting to no policy
I20260927 15:10:18.899819 22828 heartbeater.cc:347] Connected to a master server at 127.7.17.254:46765
I20260927 15:10:18.900048 22828 heartbeater.cc:464] Registering TS with master...
I20260927 15:10:18.900595 22828 heartbeater.cc:511] Master 127.7.17.254:46765 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.194@KRBTEST.COM" created.
I20260927 15:10:18.902482 22637 ts_manager.cc:194] Registered new tserver with Master: 0a1b0c9c4d49448f9d40a69effa00d21 (127.7.17.193:32919)
I20260927 15:10:18.903911 22637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:52571
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
I20260927 15:10:18.919620 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46765
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:10:19.029873 22833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:19.030153 22833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:19.030215 22833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:19.034323 22833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:19.034403 22833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:19.034490 22833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:10:19.039968 22833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:10:19.041131 22833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:19.042133 22833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:10:19.051965 22833 server_base.cc:1034] running on GCE node
W20260927 15:10:19.051920 22841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:19.051909 22838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:19.051903 22839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:19.052562 22833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:19.053227 22833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:19.054394 22833 hybrid_clock.cc:648] HybridClock initialized: now 1790521819054364 us; error 41 us; skew 500 ppm
Sep 27 15:10:19 dist-test-slave-hhpq krb5kdc[22592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521819, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:19.057133 22833 init.cc:377] Logged in from keytab as kudu/127.7.17.194@KRBTEST.COM (short username kudu)
I20260927 15:10:19.058037 22833 webserver.cc:460] Webserver started at http://127.7.17.194:45353/ using document root <none> and password file <none>
I20260927 15:10:19.058588 22833 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:19.058671 22833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:19.058879 22833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:10:19.061000 22833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "9a560782e4d14bf2a19685db01492181"
format_stamp: "Formatted at 2026-09-27 15:10:19 on dist-test-slave-hhpq"
server_key: "6436e3aa564b581374fa03abac4e10bc"
server_key_iv: "58fe4fcb26abdbb22d3e8fc8aa9e4f24"
server_key_version: "encryptionkey@0"
I20260927 15:10:19.061560 22833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "9a560782e4d14bf2a19685db01492181"
format_stamp: "Formatted at 2026-09-27 15:10:19 on dist-test-slave-hhpq"
server_key: "6436e3aa564b581374fa03abac4e10bc"
server_key_iv: "58fe4fcb26abdbb22d3e8fc8aa9e4f24"
server_key_version: "encryptionkey@0"
I20260927 15:10:19.065383 22833 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:10:19.068171 22848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:19.069693 22833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:10:19.069829 22833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "9a560782e4d14bf2a19685db01492181"
format_stamp: "Formatted at 2026-09-27 15:10:19 on dist-test-slave-hhpq"
server_key: "6436e3aa564b581374fa03abac4e10bc"
server_key_iv: "58fe4fcb26abdbb22d3e8fc8aa9e4f24"
server_key_version: "encryptionkey@0"
I20260927 15:10:19.069947 22833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:19.083295 22833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:19.086522 22833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:19.086701 22833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:19.087278 22833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:19.088127 22833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:10:19.088173 22833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:19.088213 22833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:10:19.088232 22833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:19.096740 22833 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:39193
I20260927 15:10:19.096782 22961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:39193 every 8 connection(s)
I20260927 15:10:19.097728 22833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:10:19.106309 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 22833
I20260927 15:10:19.106427 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:10:19.106695 7239 external_mini_cluster.cc:1428] Setting key 4e1cc9807c6172395ed0298186643a96
Sep 27 15:10:19 dist-test-slave-hhpq krb5kdc[22592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521819, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:10:19.112229 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:19.099791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:40597 (local address 127.7.17.254:46765)
0927 15:10:19.099945 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:19.099949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:19.100879 (+ 930us) server_negotiation.cc:407] Connection header received
0927 15:10:19.102336 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:19.102339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:19.102390 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:19.102474 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:19.103729 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.104301 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.105525 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.105781 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.105874 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:19.108694 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:19.108697 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:19.108698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:19.108722 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:19.110230 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:19.110737 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:19.110740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:19.110741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:19.110819 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:19.111141 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:19.111144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:19.111145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:19.111319 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:19.111408 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:19.111940 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 15:10:19.112066 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:10:19.113195 22964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:19.100202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:40597)
0927 15:10:19.100714 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:19.100765 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:19.102084 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:19.102673 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:19.102681 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:19.103051 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:10:19.103562 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:19.103579 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.104405 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.104407 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:19.105392 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:19.105403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.105918 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.105920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:19.106115 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:19.106779 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:10:19.106803 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:10:19.108522 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:10:19.110350 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:19.110356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:19.110373 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:19.110641 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:19.110898 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:19.110901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:19.110903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:19.111055 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:19.111413 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:10:19.111419 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:10:19.111706 (+ 287us) client_negotiation.cc:769] Sending connection context
0927 15:10:19.111879 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 15:10:19.112228 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":115,"threads_started":1}
I20260927 15:10:19.114341 22962 heartbeater.cc:347] Connected to a master server at 127.7.17.254:46765
I20260927 15:10:19.114629 22962 heartbeater.cc:464] Registering TS with master...
I20260927 15:10:19.115252 22962 heartbeater.cc:511] Master 127.7.17.254:46765 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.17.195@KRBTEST.COM; defaulting to no policy
I20260927 15:10:19.116463 22637 ts_manager.cc:194] Registered new tserver with Master: 9a560782e4d14bf2a19685db01492181 (127.7.17.194:39193)
I20260927 15:10:19.117105 22637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:40597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
I20260927 15:10:19.135659 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46765
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:10:19.253562 22967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:19.253954 22967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:19.254029 22967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:19.260264 22967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:19.260363 22967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:19.260455 22967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:10:19.268795 22967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:10:19.269989 22967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:19.270982 22967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:19.280643 22973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:19.280616 22972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:19.280727 22967 server_base.cc:1034] running on GCE node
W20260927 15:10:19.280637 22975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:19.281363 22967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:19.282078 22967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:19.283247 22967 hybrid_clock.cc:648] HybridClock initialized: now 1790521819283219 us; error 48 us; skew 500 ppm
Sep 27 15:10:19 dist-test-slave-hhpq krb5kdc[22592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521819, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:19.285992 22967 init.cc:377] Logged in from keytab as kudu/127.7.17.195@KRBTEST.COM (short username kudu)
I20260927 15:10:19.286903 22967 webserver.cc:460] Webserver started at http://127.7.17.195:45207/ using document root <none> and password file <none>
I20260927 15:10:19.287478 22967 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:19.287564 22967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:19.287773 22967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:10:19.289912 22967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "6b9e6dcbb0844c53976e942ccdd36229"
format_stamp: "Formatted at 2026-09-27 15:10:19 on dist-test-slave-hhpq"
server_key: "8285cb926540b1ae4d2e3ff10ab9f668"
server_key_iv: "35a1cbd1e3b68f72d4a1b13c5f07cd8c"
server_key_version: "encryptionkey@0"
I20260927 15:10:19.290472 22967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "6b9e6dcbb0844c53976e942ccdd36229"
format_stamp: "Formatted at 2026-09-27 15:10:19 on dist-test-slave-hhpq"
server_key: "8285cb926540b1ae4d2e3ff10ab9f668"
server_key_iv: "35a1cbd1e3b68f72d4a1b13c5f07cd8c"
server_key_version: "encryptionkey@0"
I20260927 15:10:19.294417 22967 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:10:19.297202 22982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:19.298558 22967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:10:19.298698 22967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "6b9e6dcbb0844c53976e942ccdd36229"
format_stamp: "Formatted at 2026-09-27 15:10:19 on dist-test-slave-hhpq"
server_key: "8285cb926540b1ae4d2e3ff10ab9f668"
server_key_iv: "35a1cbd1e3b68f72d4a1b13c5f07cd8c"
server_key_version: "encryptionkey@0"
I20260927 15:10:19.298817 22967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:19.311769 22967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:19.314821 22967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:19.314996 22967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:19.315577 22967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:19.316408 22967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:10:19.316455 22967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:19.316509 22967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:10:19.316537 22967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:19.324942 22967 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:38749
I20260927 15:10:19.324971 23095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:38749 every 8 connection(s)
I20260927 15:10:19.325831 22967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:10:19.332078 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 22967
I20260927 15:10:19.332216 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:10:19.332484 7239 external_mini_cluster.cc:1428] Setting key a8afe1b84f6a9b84670415db2093dc42
Sep 27 15:10:19 dist-test-slave-hhpq krb5kdc[22592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521819, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:10:19.340026 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:19.327694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:52065 (local address 127.7.17.254:46765)
0927 15:10:19.327839 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:19.327843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:19.328797 (+ 954us) server_negotiation.cc:407] Connection header received
0927 15:10:19.330072 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:19.330075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:19.330135 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:19.330209 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:19.331434 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.332100 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.333582 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.333867 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.333967 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:19.336466 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:19.336469 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:19.336471 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:19.336498 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:19.337862 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:19.338422 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:19.338425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:19.338426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:19.338510 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:19.338894 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:19.338897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:19.338898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:19.339107 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:19.339190 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:19.339772 (+ 582us) server_negotiation.cc:299] Negotiation successful
0927 15:10:19.339897 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:10:19.340890 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:19.328175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.195:52065)
0927 15:10:19.328633 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:19.328679 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:19.329858 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:19.330376 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:19.330383 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:19.330762 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:10:19.331279 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:19.331293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.332250 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.332253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:19.333468 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:19.333480 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.334018 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.334022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:19.334213 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:19.334797 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:10:19.334813 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:10:19.336298 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:10:19.337984 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:19.337989 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:19.338003 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:19.338290 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:19.338635 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:19.338638 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:19.338639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:19.338790 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:19.339223 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:10:19.339229 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:10:19.339485 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 15:10:19.339709 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 15:10:19.339987 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":105,"threads_started":1}
I20260927 15:10:19.342151 23096 heartbeater.cc:347] Connected to a master server at 127.7.17.254:46765
I20260927 15:10:19.342384 23096 heartbeater.cc:464] Registering TS with master...
I20260927 15:10:19.342919 23096 heartbeater.cc:511] Master 127.7.17.254:46765 requested a full tablet report, sending...
I20260927 15:10:19.344045 22637 ts_manager.cc:194] Registered new tserver with Master: 6b9e6dcbb0844c53976e942ccdd36229 (127.7.17.195:38749)
I20260927 15:10:19.344646 22637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:52065
I20260927 15:10:19.346570 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:10:19.355278 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:19.348428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37420 (local address 127.7.17.254:46765)
0927 15:10:19.348540 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:19.348566 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:19.348684 (+ 118us) server_negotiation.cc:407] Connection header received
0927 15:10:19.348801 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:19.348804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:19.348867 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:19.348937 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:19.349626 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.350310 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.351208 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.351417 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.351495 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:19.352669 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:19.352672 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:19.352674 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:19.352695 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:19.353999 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:19.354361 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:19.354364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:19.354365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:19.354431 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:19.354612 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:19.354614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:19.354615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:19.354814 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:19.354910 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:19.355066 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 15:10:19.355157 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 15:10:19.358222 22637 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:10:19.361274 22637 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:10:19.376070 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:19.371294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56732 (local address 127.7.17.194:39193)
0927 15:10:19.371705 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:19.371725 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:19.371751 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:10:19.371828 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:19.371833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:19.371992 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:19.372129 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:19.372515 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.373559 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.374696 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.375387 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.375485 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:19.375649 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:19.375818 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 15:10:19.375910 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":113,"threads_started":1}
I20260927 15:10:19.376060 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:19.370893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:39193 (local address 127.0.0.1:56732)
0927 15:10:19.371406 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:19.371418 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:19.371491 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:19.372284 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:19.372287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:19.372304 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:19.372389 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:19.372395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.373676 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.373680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:19.374571 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:19.374579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.375506 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.375510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:19.375614 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:19.375664 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 15:10:19.375776 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 15:10:19.375907 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":81,"threads_started":1}
I20260927 15:10:19.376655 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:19.370121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:38749 (local address 127.0.0.1:35218)
0927 15:10:19.371115 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:19.371128 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:19.371225 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:19.371843 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:19.371849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:19.371870 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:19.372008 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:19.372016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.373409 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.373412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:19.375085 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:19.375094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.376310 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.376314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:19.376395 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:19.376414 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 15:10:19.376466 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 15:10:19.376520 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"mutex_wait_us":49,"thread_start_us":93,"threads_started":1}
I20260927 15:10:19.376718 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:19.370256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.7.17.193:32919)
0927 15:10:19.370667 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:19.370682 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:19.371208 (+ 526us) server_negotiation.cc:407] Connection header received
0927 15:10:19.371511 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:19.371517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:19.371706 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:19.371841 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:19.372308 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.373218 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.374913 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.375729 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.375831 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:19.375968 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:19.376465 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 15:10:19.376573 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":65,"threads_started":1}
I20260927 15:10:19.377136 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:19.370464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35218 (local address 127.7.17.195:38749)
0927 15:10:19.370857 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:19.370869 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:19.371338 (+ 469us) server_negotiation.cc:407] Connection header received
0927 15:10:19.371424 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:19.371431 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:19.371679 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:19.371841 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:19.372144 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.373274 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.375217 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.376026 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.376121 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:19.376252 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:19.376932 (+ 680us) server_negotiation.cc:299] Negotiation successful
0927 15:10:19.377026 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":164,"threads_started":1}
I20260927 15:10:19.376655 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:19.370085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:32919 (local address 127.0.0.1:38584)
0927 15:10:19.370998 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:19.371053 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:19.371196 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:19.372096 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:19.372098 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:19.372116 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:19.372199 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:19.372204 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.373335 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.373338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:19.374785 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:19.374795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.376035 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.376038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:19.376137 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:19.376363 (+ 226us) client_negotiation.cc:769] Sending connection context
0927 15:10:19.376424 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 15:10:19.376497 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":95,"thread_start_us":118,"threads_started":1}
I20260927 15:10:19.379247 22896 tablet_service.cc:1467] Processing CreateTablet for tablet 021a5875f67b4166977bb21371dc7734 (DEFAULT_TABLE table=test-table [id=819b7214b8a04aa29f791946b64f5d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:10:19.380240 22896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 021a5875f67b4166977bb21371dc7734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:10:19.379992 23030 tablet_service.cc:1467] Processing CreateTablet for tablet 021a5875f67b4166977bb21371dc7734 (DEFAULT_TABLE table=test-table [id=819b7214b8a04aa29f791946b64f5d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:10:19.379995 22762 tablet_service.cc:1467] Processing CreateTablet for tablet 021a5875f67b4166977bb21371dc7734 (DEFAULT_TABLE table=test-table [id=819b7214b8a04aa29f791946b64f5d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:10:19.381071 22762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 021a5875f67b4166977bb21371dc7734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:10:19.381085 23030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 021a5875f67b4166977bb21371dc7734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:10:19.387336 23113 tablet_bootstrap.cc:492] T 021a5875f67b4166977bb21371dc7734 P 9a560782e4d14bf2a19685db01492181: Bootstrap starting.
I20260927 15:10:19.388049 23114 tablet_bootstrap.cc:492] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229: Bootstrap starting.
I20260927 15:10:19.389293 23115 tablet_bootstrap.cc:492] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21: Bootstrap starting.
I20260927 15:10:19.390091 23113 tablet_bootstrap.cc:654] T 021a5875f67b4166977bb21371dc7734 P 9a560782e4d14bf2a19685db01492181: Neither blocks nor log segments found. Creating new log.
I20260927 15:10:19.390667 23114 tablet_bootstrap.cc:654] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229: Neither blocks nor log segments found. Creating new log.
I20260927 15:10:19.390996 23115 tablet_bootstrap.cc:654] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21: Neither blocks nor log segments found. Creating new log.
I20260927 15:10:19.391018 23113 log.cc:824] T 021a5875f67b4166977bb21371dc7734 P 9a560782e4d14bf2a19685db01492181: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:19.391516 23114 log.cc:824] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:19.392046 23115 log.cc:824] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:19.393435 23113 tablet_bootstrap.cc:492] T 021a5875f67b4166977bb21371dc7734 P 9a560782e4d14bf2a19685db01492181: No bootstrap required, opened a new log
I20260927 15:10:19.393690 23113 ts_tablet_manager.cc:1397] T 021a5875f67b4166977bb21371dc7734 P 9a560782e4d14bf2a19685db01492181: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:10:19.393739 23114 tablet_bootstrap.cc:492] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229: No bootstrap required, opened a new log
I20260927 15:10:19.393826 23115 tablet_bootstrap.cc:492] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21: No bootstrap required, opened a new log
I20260927 15:10:19.393894 23114 ts_tablet_manager.cc:1397] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260927 15:10:19.394006 23115 ts_tablet_manager.cc:1397] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 15:10:19.396765 23115 raft_consensus.cc:348] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } } peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } }
I20260927 15:10:19.397073 23115 raft_consensus.cc:374] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:10:19.397159 23115 raft_consensus.cc:729] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a1b0c9c4d49448f9d40a69effa00d21, State: Initialized, Role: FOLLOWER
I20260927 15:10:19.397617 23115 consensus_queue.cc:260] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } } peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } }
I20260927 15:10:19.397645 23113 raft_consensus.cc:348] T 021a5875f67b4166977bb21371dc7734 P 9a560782e4d14bf2a19685db01492181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } } peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } }
I20260927 15:10:19.397639 23114 raft_consensus.cc:348] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } } peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } }
I20260927 15:10:19.397922 23113 raft_consensus.cc:374] T 021a5875f67b4166977bb21371dc7734 P 9a560782e4d14bf2a19685db01492181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:10:19.397922 23114 raft_consensus.cc:374] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:10:19.397979 23114 raft_consensus.cc:729] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9e6dcbb0844c53976e942ccdd36229, State: Initialized, Role: FOLLOWER
I20260927 15:10:19.397987 23113 raft_consensus.cc:729] T 021a5875f67b4166977bb21371dc7734 P 9a560782e4d14bf2a19685db01492181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a560782e4d14bf2a19685db01492181, State: Initialized, Role: FOLLOWER
I20260927 15:10:19.398443 23114 consensus_queue.cc:260] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } } peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } }
I20260927 15:10:19.398443 23113 consensus_queue.cc:260] T 021a5875f67b4166977bb21371dc7734 P 9a560782e4d14bf2a19685db01492181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } } peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } }
I20260927 15:10:19.398624 23115 ts_tablet_manager.cc:1428] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 15:10:19.399025 22828 heartbeater.cc:503] Master 127.7.17.254:46765 was elected leader, sending a full tablet report...
I20260927 15:10:19.399127 23096 heartbeater.cc:503] Master 127.7.17.254:46765 was elected leader, sending a full tablet report...
I20260927 15:10:19.400044 22962 heartbeater.cc:503] Master 127.7.17.254:46765 was elected leader, sending a full tablet report...
I20260927 15:10:19.400255 23114 ts_tablet_manager.cc:1428] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:10:19.401212 23113 ts_tablet_manager.cc:1428] T 021a5875f67b4166977bb21371dc7734 P 9a560782e4d14bf2a19685db01492181: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260927 15:10:19.576872 23097 tablet.cc:2369] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:10:19.599180 22963 tablet.cc:2369] T 021a5875f67b4166977bb21371dc7734 P 9a560782e4d14bf2a19685db01492181: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:10:19.613505 23120 raft_consensus.cc:482] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:10:19.613719 23120 raft_consensus.cc:504] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } } peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } }
I20260927 15:10:19.614935 23120 leader_election.cc:288] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a1b0c9c4d49448f9d40a69effa00d21 (127.7.17.193:32919), 9a560782e4d14bf2a19685db01492181 (127.7.17.194:39193)
I20260927 15:10:19.618604 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:19.615342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:52495 (local address 127.7.17.194:39193)
0927 15:10:19.615464 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:19.615468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:19.615484 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:10:19.615646 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:19.615649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:19.615695 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:19.615773 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:19.616066 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.616578 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.617665 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.618075 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.618190 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:19.618248 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:19.618409 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 15:10:19.618487 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:10:19.618605 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:19.615254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:39193 (local address 127.7.17.195:52495)
0927 15:10:19.615371 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:19.615386 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:19.615492 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:19.615813 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:19.615816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:19.615837 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:19.615943 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:19.615949 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.616692 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.616696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:19.617530 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:19.617540 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.618190 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.618194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:19.618297 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:19.618317 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 15:10:19.618380 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 15:10:19.618458 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 15:10:19.618950 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:19.615323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:52505 (local address 127.7.17.193:32919)
0927 15:10:19.615434 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:19.615438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:19.615987 (+ 549us) server_negotiation.cc:407] Connection header received
0927 15:10:19.616083 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:19.616087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:19.616133 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:19.616193 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:19.616441 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.616965 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.618010 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.618406 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.618510 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:19.618558 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:19.618752 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 15:10:19.618836 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:10:19.619289 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:19.615258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:32919 (local address 127.7.17.195:52505)
0927 15:10:19.615894 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:19.615904 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:19.615998 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:19.616217 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:19.616220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:19.616234 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:19.616338 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:19.616343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.617072 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.617075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:19.617884 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:19.617893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.618515 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.618518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:19.618617 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:19.618635 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 15:10:19.619099 (+ 464us) client_negotiation.cc:240] Negotiation successful
0927 15:10:19.619177 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":306,"thread_start_us":102,"threads_started":1}
I20260927 15:10:19.619321 22916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021a5875f67b4166977bb21371dc7734" candidate_uuid: "6b9e6dcbb0844c53976e942ccdd36229" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a560782e4d14bf2a19685db01492181" is_pre_election: true
I20260927 15:10:19.619704 22916 raft_consensus.cc:2454] T 021a5875f67b4166977bb21371dc7734 P 9a560782e4d14bf2a19685db01492181 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b9e6dcbb0844c53976e942ccdd36229 in term 0.
I20260927 15:10:19.619915 22782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021a5875f67b4166977bb21371dc7734" candidate_uuid: "6b9e6dcbb0844c53976e942ccdd36229" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" is_pre_election: true
I20260927 15:10:19.620198 22782 raft_consensus.cc:2454] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b9e6dcbb0844c53976e942ccdd36229 in term 0.
I20260927 15:10:19.620193 22986 leader_election.cc:302] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b9e6dcbb0844c53976e942ccdd36229, 9a560782e4d14bf2a19685db01492181; no voters:
I20260927 15:10:19.620443 23120 raft_consensus.cc:2781] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:10:19.620530 23120 raft_consensus.cc:482] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:10:19.620574 23120 raft_consensus.cc:3037] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:10:19.621647 23120 raft_consensus.cc:504] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } } peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } }
I20260927 15:10:19.621982 23120 leader_election.cc:288] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229 [CANDIDATE]: Term 1 election: Requested vote from peers 0a1b0c9c4d49448f9d40a69effa00d21 (127.7.17.193:32919), 9a560782e4d14bf2a19685db01492181 (127.7.17.194:39193)
I20260927 15:10:19.622319 22916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021a5875f67b4166977bb21371dc7734" candidate_uuid: "6b9e6dcbb0844c53976e942ccdd36229" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a560782e4d14bf2a19685db01492181"
I20260927 15:10:19.622344 22782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021a5875f67b4166977bb21371dc7734" candidate_uuid: "6b9e6dcbb0844c53976e942ccdd36229" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a1b0c9c4d49448f9d40a69effa00d21"
I20260927 15:10:19.622442 22916 raft_consensus.cc:3037] T 021a5875f67b4166977bb21371dc7734 P 9a560782e4d14bf2a19685db01492181 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:10:19.622447 22782 raft_consensus.cc:3037] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:10:19.623508 22916 raft_consensus.cc:2454] T 021a5875f67b4166977bb21371dc7734 P 9a560782e4d14bf2a19685db01492181 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b9e6dcbb0844c53976e942ccdd36229 in term 1.
I20260927 15:10:19.623566 22782 raft_consensus.cc:2454] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b9e6dcbb0844c53976e942ccdd36229 in term 1.
I20260927 15:10:19.623806 22986 leader_election.cc:302] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b9e6dcbb0844c53976e942ccdd36229, 9a560782e4d14bf2a19685db01492181; no voters:
I20260927 15:10:19.623967 23120 raft_consensus.cc:2781] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:10:19.624197 23120 raft_consensus.cc:686] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229 [term 1 LEADER]: Becoming Leader. State: Replica: 6b9e6dcbb0844c53976e942ccdd36229, State: Running, Role: LEADER
I20260927 15:10:19.624537 23120 consensus_queue.cc:237] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } } peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } }
I20260927 15:10:19.627902 22637 catalog_manager.cc:5495] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b9e6dcbb0844c53976e942ccdd36229 (127.7.17.195). New cstate: current_term: 1 leader_uuid: "6b9e6dcbb0844c53976e942ccdd36229" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } health_report { overall_health: UNKNOWN } } }
W20260927 15:10:19.634013 22829 tablet.cc:2369] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:10:19.670382 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:19.666679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35222 (local address 127.7.17.195:38749)
0927 15:10:19.666813 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:19.666817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:19.666830 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:10:19.666906 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:19.666909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:19.666966 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:19.667067 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:10:19.667419 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.667974 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.668854 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:19.669068 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:19.669148 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:19.669462 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:10:19.669940 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:10:19.670043 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:19.670153 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 15:10:19.670213 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:10:19.676529 22782 raft_consensus.cc:1261] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 1 FOLLOWER]: Refusing update from remote peer 6b9e6dcbb0844c53976e942ccdd36229: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:10:19.676529 22916 raft_consensus.cc:1261] T 021a5875f67b4166977bb21371dc7734 P 9a560782e4d14bf2a19685db01492181 [term 1 FOLLOWER]: Refusing update from remote peer 6b9e6dcbb0844c53976e942ccdd36229: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:10:19.677275 23123 consensus_queue.cc:1035] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:10:19.677469 23120 consensus_queue.cc:1035] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:10:19.684266 23126 mvcc.cc:205] Tried to move back new op lower bound from 7333977373387677696 to 7333977373184798720. Current Snapshot: MvccSnapshot[applied={T|T < 7333977373387677696}]
I20260927 15:10:19.687053 23127 mvcc.cc:205] Tried to move back new op lower bound from 7333977373387677696 to 7333977373184798720. Current Snapshot: MvccSnapshot[applied={T|T < 7333977373387677696}]
I20260927 15:10:19.693403 22637 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:10:19.693575 22637 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:10:19.696195 22637 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b: Sending DeleteTablet for 3 replicas of tablet 021a5875f67b4166977bb21371dc7734
I20260927 15:10:19.697170 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 22699
I20260927 15:10:19.697129 22762 tablet_service.cc:1514] Processing DeleteTablet for tablet 021a5875f67b4166977bb21371dc7734 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:10:19 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:38584
I20260927 15:10:19.697150 23030 tablet_service.cc:1514] Processing DeleteTablet for tablet 021a5875f67b4166977bb21371dc7734 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:10:19 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:35218
I20260927 15:10:19.697837 22896 tablet_service.cc:1514] Processing DeleteTablet for tablet 021a5875f67b4166977bb21371dc7734 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:10:19 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:56732
I20260927 15:10:19.697961 23135 tablet_replica.cc:330] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229: stopping tablet replica
I20260927 15:10:19.698271 23136 tablet_replica.cc:330] T 021a5875f67b4166977bb21371dc7734 P 9a560782e4d14bf2a19685db01492181: stopping tablet replica
I20260927 15:10:19.698318 23135 raft_consensus.cc:2229] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:10:19.698553 23136 raft_consensus.cc:2229] T 021a5875f67b4166977bb21371dc7734 P 9a560782e4d14bf2a19685db01492181 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:10:19.698807 23135 raft_consensus.cc:2258] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:10:19.698890 23136 raft_consensus.cc:2258] T 021a5875f67b4166977bb21371dc7734 P 9a560782e4d14bf2a19685db01492181 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:10:19.700537 23135 ts_tablet_manager.cc:1905] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:10:19.700567 23136 ts_tablet_manager.cc:1905] T 021a5875f67b4166977bb21371dc7734 P 9a560782e4d14bf2a19685db01492181: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:10:19.703662 23136 ts_tablet_manager.cc:1918] T 021a5875f67b4166977bb21371dc7734 P 9a560782e4d14bf2a19685db01492181: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:10:19.703837 23136 log.cc:1196] T 021a5875f67b4166977bb21371dc7734 P 9a560782e4d14bf2a19685db01492181: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/wal/wals/021a5875f67b4166977bb21371dc7734
I20260927 15:10:19.703958 23135 ts_tablet_manager.cc:1918] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:10:19.704124 23135 log.cc:1196] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/021a5875f67b4166977bb21371dc7734
I20260927 15:10:19.704180 23136 ts_tablet_manager.cc:1939] T 021a5875f67b4166977bb21371dc7734 P 9a560782e4d14bf2a19685db01492181: Deleting consensus metadata
I20260927 15:10:19.704475 23135 ts_tablet_manager.cc:1939] T 021a5875f67b4166977bb21371dc7734 P 6b9e6dcbb0844c53976e942ccdd36229: Deleting consensus metadata
I20260927 15:10:19.705888 22624 catalog_manager.cc:4841] TS 6b9e6dcbb0844c53976e942ccdd36229 (127.7.17.195:38749): tablet 021a5875f67b4166977bb21371dc7734 (table test-table [id=819b7214b8a04aa29f791946b64f5d9d]) successfully deleted
I20260927 15:10:19.706220 22624 catalog_manager.cc:4841] TS 9a560782e4d14bf2a19685db01492181 (127.7.17.194:39193): tablet 021a5875f67b4166977bb21371dc7734 (table test-table [id=819b7214b8a04aa29f791946b64f5d9d]) successfully deleted
W20260927 15:10:19.706808 22622 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.17.193:32919 (error 108)
I20260927 15:10:19.707026 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 22833
I20260927 15:10:19.708680 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:19.708301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:32919 (local address 127.0.0.1:38586)
0927 15:10:19.708505 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:19.708585 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.193:32919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260927 15:10:19.708860 22622 catalog_manager.cc:4584] TS 0a1b0c9c4d49448f9d40a69effa00d21 (127.7.17.193:32919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 021a5875f67b4166977bb21371dc7734: Network error: Client connection negotiation failed: client connection to 127.7.17.193:32919: connect: Connection refused (error 111)
I20260927 15:10:19.712911 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 22967
I20260927 15:10:19.718123 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 22605
I20260927 15:10:39.724263 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:46765
--webserver_interface=127.7.17.254
--webserver_port=36883
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:46765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:10:39.841557 23138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:39.841863 23138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:39.841921 23138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:39.846110 23138 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 15:10:39.846189 23138 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 15:10:39.846210 23138 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 15:10:39.846227 23138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:10:39.846249 23138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:39.846274 23138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:39.846303 23138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:10:39.846326 23138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:10:39.852169 23138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:46765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:46765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=36883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:10:39.853410 23138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:39.854504 23138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:39.867692 23145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:39.867697 23144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:39.867789 23138 server_base.cc:1034] running on GCE node
W20260927 15:10:39.867693 23147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:39.868661 23138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:39.869933 23138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:39.871110 23138 hybrid_clock.cc:648] HybridClock initialized: now 1790521839871087 us; error 36 us; skew 500 ppm
Sep 27 15:10:39 dist-test-slave-hhpq krb5kdc[22592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521839, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:39.874186 23138 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:10:39.875236 23138 webserver.cc:460] Webserver started at http://127.7.17.254:36883/ using document root <none> and password file <none>
I20260927 15:10:39.875839 23138 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:39.875918 23138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:39.881208 23138 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:10:39.883805 23154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:39.885331 23138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:10:39.885483 23138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "af580a3e658c4e8a9731da421183718b"
format_stamp: "Formatted at 2026-09-27 15:10:18 on dist-test-slave-hhpq"
server_key: "1364626bf50f3700bf560ab9017792db"
server_key_iv: "b4bda720f322c0eb339345d71e7d85fe"
server_key_version: "encryptionkey@0"
I20260927 15:10:39.886188 23138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:39.932330 23138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:39.936224 23138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:39.936431 23138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:39.944383 23138 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:46765
I20260927 15:10:39.944708 23206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:46765 every 8 connection(s)
I20260927 15:10:39.945869 23138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:10:39.950887 23207 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:10:39.951678 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 23138
I20260927 15:10:39.952183 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:32919
--local_ip_for_outbound_sockets=127.7.17.193
--tserver_master_addrs=127.7.17.254:46765
--webserver_port=36101
--webserver_interface=127.7.17.193
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:10:39.953372 23207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b: Bootstrap starting.
I20260927 15:10:39.971120 23207 log.cc:824] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:39.979930 23207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:10:39.980304 23207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b: Bootstrap complete.
I20260927 15:10:39.984206 23207 raft_consensus.cc:348] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af580a3e658c4e8a9731da421183718b" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46765 } }
I20260927 15:10:39.984640 23207 raft_consensus.cc:729] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: af580a3e658c4e8a9731da421183718b, State: Initialized, Role: FOLLOWER
I20260927 15:10:39.985208 23207 consensus_queue.cc:260] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af580a3e658c4e8a9731da421183718b" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46765 } }
I20260927 15:10:39.985343 23207 raft_consensus.cc:388] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:10:39.985420 23207 raft_consensus.cc:482] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:10:39.985522 23207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:10:39.987622 23207 raft_consensus.cc:504] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af580a3e658c4e8a9731da421183718b" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46765 } }
I20260927 15:10:39.987975 23207 leader_election.cc:302] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af580a3e658c4e8a9731da421183718b; no voters:
I20260927 15:10:39.988279 23207 leader_election.cc:288] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:10:39.988431 23211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:10:39.988690 23211 raft_consensus.cc:686] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 2 LEADER]: Becoming Leader. State: Replica: af580a3e658c4e8a9731da421183718b, State: Running, Role: LEADER
I20260927 15:10:39.989071 23211 consensus_queue.cc:237] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af580a3e658c4e8a9731da421183718b" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46765 } }
I20260927 15:10:39.989557 23207 sys_catalog.cc:564] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:10:39.990552 23213 sys_catalog.cc:455] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [sys.catalog]: SysCatalogTable state changed. Reason: New leader af580a3e658c4e8a9731da421183718b. Latest consensus state: current_term: 2 leader_uuid: "af580a3e658c4e8a9731da421183718b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af580a3e658c4e8a9731da421183718b" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46765 } } }
I20260927 15:10:39.990682 23213 sys_catalog.cc:458] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [sys.catalog]: This master's current role is: LEADER
I20260927 15:10:39.991031 23212 sys_catalog.cc:455] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "af580a3e658c4e8a9731da421183718b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af580a3e658c4e8a9731da421183718b" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46765 } } }
I20260927 15:10:39.991114 23212 sys_catalog.cc:458] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [sys.catalog]: This master's current role is: LEADER
I20260927 15:10:39.994531 23227 catalog_manager.cc:1259] Loaded cluster ID: 936dd6b417434a0c971f8674f81c4d4a
I20260927 15:10:39.994601 23227 catalog_manager.cc:1552] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b: loading cluster ID for follower catalog manager: success
I20260927 15:10:39.996421 23227 catalog_manager.cc:1574] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b: acquiring CA information for follower catalog manager: success
I20260927 15:10:39.997947 23227 catalog_manager.cc:1602] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:10:39.998096 23220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:10:40.000623 23220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:10:40.001142 23220 catalog_manager.cc:1259] Loaded cluster ID: 936dd6b417434a0c971f8674f81c4d4a
I20260927 15:10:40.001204 23220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:10:40.001899 23220 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:10:40.002386 23220 catalog_manager.cc:5879] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b: Loaded TSK: 0
I20260927 15:10:40.017922 23220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b: Generated new TSK 1
I20260927 15:10:40.018229 23220 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:10:40.075730 23209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:40.076026 23209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:40.076074 23209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:40.080480 23209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:40.080561 23209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:40.080627 23209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:10:40.086244 23209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:32919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=36101
--tserver_master_addrs=127.7.17.254:46765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:10:40.087328 23209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:40.088436 23209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:40.099673 23235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:40.099690 23234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:40.099673 23237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:40.100109 23209 server_base.cc:1034] running on GCE node
I20260927 15:10:40.100689 23209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:40.101433 23209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:40.102612 23209 hybrid_clock.cc:648] HybridClock initialized: now 1790521840102583 us; error 44 us; skew 500 ppm
Sep 27 15:10:40 dist-test-slave-hhpq krb5kdc[22592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521840, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:40.105664 23209 init.cc:377] Logged in from keytab as kudu/127.7.17.193@KRBTEST.COM (short username kudu)
I20260927 15:10:40.106621 23209 webserver.cc:460] Webserver started at http://127.7.17.193:36101/ using document root <none> and password file <none>
I20260927 15:10:40.107190 23209 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:40.107276 23209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:40.111631 23209 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:10:40.114225 23244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:40.115890 23209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:10:40.116129 23209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "0a1b0c9c4d49448f9d40a69effa00d21"
format_stamp: "Formatted at 2026-09-27 15:10:18 on dist-test-slave-hhpq"
server_key: "47927d8866c8b84da96eec2b19045bf3"
server_key_iv: "d7b72d30db6d4f10bde001e5f95ae503"
server_key_version: "encryptionkey@0"
I20260927 15:10:40.116873 23209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:40.132723 23209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:40.136365 23209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:40.136577 23209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:40.137271 23209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:40.138388 23251 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 15:10:40.142900 23209 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 15:10:40.142987 23209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 15:10:40.143060 23209 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 15:10:40.144953 23209 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 15:10:40.145011 23209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260927 15:10:40.145318 23251 tablet_bootstrap.cc:492] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21: Bootstrap starting.
I20260927 15:10:40.154763 23209 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:32919
I20260927 15:10:40.154875 23358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:32919 every 8 connection(s)
I20260927 15:10:40.155767 23209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:10:40.159729 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 23209
I20260927 15:10:40.160147 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:39193
--local_ip_for_outbound_sockets=127.7.17.194
--tserver_master_addrs=127.7.17.254:46765
--webserver_port=45353
--webserver_interface=127.7.17.194
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:10:40.163584 23251 log.cc:824] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:10:40 dist-test-slave-hhpq krb5kdc[22592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521840, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:10:40.171432 23251 tablet_bootstrap.cc:492] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:10:40.171880 23251 tablet_bootstrap.cc:492] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21: Bootstrap complete.
I20260927 15:10:40.172905 23251 ts_tablet_manager.cc:1397] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21: Time spent bootstrapping tablet: real 0.028s user 0.023s sys 0.004s
I20260927 15:10:40.174624 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:40.157876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:52853 (local address 127.7.17.254:46765)
0927 15:10:40.158250 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:40.158271 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:40.158602 (+ 331us) server_negotiation.cc:407] Connection header received
0927 15:10:40.159855 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:40.159877 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:40.160357 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:40.160768 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:40.162085 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.163071 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.164384 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.165130 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.165311 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:40.168522 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:40.168529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:40.168548 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:40.168583 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:40.170840 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:40.171557 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:40.171564 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:40.171570 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:40.171747 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:40.172161 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:40.172164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:40.172166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:40.172642 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:40.172773 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:40.173470 (+ 697us) server_negotiation.cc:299] Negotiation successful
0927 15:10:40.173779 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":112,"threads_started":1}
I20260927 15:10:40.174645 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:40.157750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.193:52853)
0927 15:10:40.158413 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:40.158471 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:40.159293 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:40.160887 (+ 1594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:40.160896 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:40.161353 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:10:40.161917 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:40.161931 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.163206 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.163209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:40.164250 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:40.164262 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.165525 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.165529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:40.165743 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:40.166428 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:10:40.166449 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:10:40.168288 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:10:40.171020 (+ 2732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:40.171026 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:40.171040 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:40.171420 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:40.171869 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:40.171872 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:40.171874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:40.172056 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:40.172757 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:10:40.172763 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:10:40.173109 (+ 346us) client_negotiation.cc:769] Sending connection context
0927 15:10:40.173307 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 15:10:40.173679 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":141,"threads_started":1}
I20260927 15:10:40.175930 23251 raft_consensus.cc:348] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } } peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } }
I20260927 15:10:40.176452 23251 raft_consensus.cc:729] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a1b0c9c4d49448f9d40a69effa00d21, State: Initialized, Role: FOLLOWER
I20260927 15:10:40.177037 23251 consensus_queue.cc:260] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } } peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } }
I20260927 15:10:40.177776 23359 heartbeater.cc:347] Connected to a master server at 127.7.17.254:46765
I20260927 15:10:40.178018 23359 heartbeater.cc:464] Registering TS with master...
I20260927 15:10:40.178143 23251 ts_tablet_manager.cc:1428] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:10:40.178573 23359 heartbeater.cc:511] Master 127.7.17.254:46765 requested a full tablet report, sending...
I20260927 15:10:40.181278 23169 ts_manager.cc:194] Registered new tserver with Master: 0a1b0c9c4d49448f9d40a69effa00d21 (127.7.17.193:32919)
I20260927 15:10:40.185093 23169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:52853
I20260927 15:10:40.187768 23359 heartbeater.cc:503] Master 127.7.17.254:46765 was elected leader, sending a full tablet report...
Sep 27 15:10:40 dist-test-slave-hhpq krb5kdc[22592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521839, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for kudu/127.7.17.193@KRBTEST.COM
I20260927 15:10:40.195636 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:40.184449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:32919 (local address 127.0.0.1:37800)
0927 15:10:40.185331 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:40.185385 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:40.185544 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:40.186148 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:40.186154 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:40.186598 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:10:40.186735 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:40.186744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.188069 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.188073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:40.189020 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:40.189030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.189845 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.189849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:40.189961 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:40.190545 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:10:40.190559 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:10:40.192154 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:10:40.193911 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:40.193922 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:40.193927 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:40.194223 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:40.194475 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:40.194478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:40.194480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:40.194576 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:40.194980 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:10:40.194986 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:10:40.195114 (+ 128us) client_negotiation.cc:769] Sending connection context
0927 15:10:40.195251 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 15:10:40.195457 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":113,"threads_started":1}
I20260927 15:10:40.195705 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:40.184575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37800 (local address 127.7.17.193:32919)
0927 15:10:40.185047 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:40.185059 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:40.185513 (+ 454us) server_negotiation.cc:407] Connection header received
0927 15:10:40.185749 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:40.185755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:40.185982 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:40.186120 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:40.186865 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.187921 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.189162 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.189722 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.189829 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:40.192302 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:40.192311 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:40.192322 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:40.192346 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:40.193806 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:40.194321 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:40.194325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:40.194327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:40.194400 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:40.194651 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:40.194653 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:40.194654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:40.194886 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:40.195007 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:40.195340 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 15:10:40.195542 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":113,"threads_started":1}
I20260927 15:10:40.197041 23293 tablet_service.cc:1514] Processing DeleteTablet for tablet 021a5875f67b4166977bb21371dc7734 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:10:19 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:37800
I20260927 15:10:40.197970 23368 tablet_replica.cc:330] stopping tablet replica
I20260927 15:10:40.198309 23368 raft_consensus.cc:2229] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:10:40.198608 23368 raft_consensus.cc:2258] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:10:40.199928 23368 ts_tablet_manager.cc:1905] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:10:40.204645 23368 ts_tablet_manager.cc:1918] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:10:40.204771 23368 log.cc:1196] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/wal/wals/021a5875f67b4166977bb21371dc7734
I20260927 15:10:40.205073 23368 ts_tablet_manager.cc:1939] T 021a5875f67b4166977bb21371dc7734 P 0a1b0c9c4d49448f9d40a69effa00d21: Deleting consensus metadata
I20260927 15:10:40.207563 23156 catalog_manager.cc:4841] TS 0a1b0c9c4d49448f9d40a69effa00d21 (127.7.17.193:32919): tablet 021a5875f67b4166977bb21371dc7734 (table test-table [id=819b7214b8a04aa29f791946b64f5d9d]) successfully deleted
W20260927 15:10:40.284021 23363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:40.284315 23363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:40.284363 23363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:40.288496 23363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:40.288573 23363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:40.288635 23363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:10:40.294318 23363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:39193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=45353
--tserver_master_addrs=127.7.17.254:46765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:10:40.295441 23363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:40.296599 23363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:10:40.306584 23376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:40.306629 23373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:40.306727 23363 server_base.cc:1034] running on GCE node
W20260927 15:10:40.306900 23374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:40.307425 23363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:40.308233 23363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:40.309418 23363 hybrid_clock.cc:648] HybridClock initialized: now 1790521840309389 us; error 40 us; skew 500 ppm
Sep 27 15:10:40 dist-test-slave-hhpq krb5kdc[22592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521840, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:40.312184 23363 init.cc:377] Logged in from keytab as kudu/127.7.17.194@KRBTEST.COM (short username kudu)
I20260927 15:10:40.313131 23363 webserver.cc:460] Webserver started at http://127.7.17.194:45353/ using document root <none> and password file <none>
I20260927 15:10:40.313670 23363 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:40.313740 23363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:40.317854 23363 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:10:40.320096 23383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:40.321655 23363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:10:40.321784 23363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "9a560782e4d14bf2a19685db01492181"
format_stamp: "Formatted at 2026-09-27 15:10:19 on dist-test-slave-hhpq"
server_key: "6436e3aa564b581374fa03abac4e10bc"
server_key_iv: "58fe4fcb26abdbb22d3e8fc8aa9e4f24"
server_key_version: "encryptionkey@0"
I20260927 15:10:40.322392 23363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:40.343021 23363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:40.346670 23363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:40.346866 23363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:40.347500 23363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:40.348384 23363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:10:40.348430 23363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:40.348484 23363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:10:40.348513 23363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:40.357460 23363 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:39193
I20260927 15:10:40.357496 23496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:39193 every 8 connection(s)
I20260927 15:10:40.358528 23363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
Sep 27 15:10:40 dist-test-slave-hhpq krb5kdc[22592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521840, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:10:40.368810 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 23363
I20260927 15:10:40.369282 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:38749
--local_ip_for_outbound_sockets=127.7.17.195
--tserver_master_addrs=127.7.17.254:46765
--webserver_port=45207
--webserver_interface=127.7.17.195
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:10:40.372229 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:40.360411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:33357 (local address 127.7.17.254:46765)
0927 15:10:40.360559 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:40.360564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:40.361486 (+ 922us) server_negotiation.cc:407] Connection header received
0927 15:10:40.362834 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:40.362838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:40.362897 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:40.362973 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:40.364240 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.364768 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.365942 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.366185 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.366283 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:40.368842 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:40.368847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:40.368850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:40.368878 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:40.370250 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:40.370745 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:40.370748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:40.370750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:40.370828 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:40.371139 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:40.371142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:40.371144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:40.371322 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:40.371402 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:40.371913 (+ 511us) server_negotiation.cc:299] Negotiation successful
0927 15:10:40.372058 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:10:40.373147 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:40.360857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:33357)
0927 15:10:40.361329 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:40.361376 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:40.362611 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:40.363201 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:40.363209 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:40.363578 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:10:40.364088 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:40.364103 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.364913 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.364916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:40.365819 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:40.365828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.366336 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.366340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:40.366502 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:40.367055 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:10:40.367080 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:10:40.368626 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:10:40.370363 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:40.370369 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:40.370386 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:40.370649 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:40.370910 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:40.370913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:40.370915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:40.371054 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:40.371416 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:10:40.371423 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:10:40.371683 (+ 260us) client_negotiation.cc:769] Sending connection context
0927 15:10:40.371858 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 15:10:40.372203 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":111,"threads_started":1}
I20260927 15:10:40.374320 23497 heartbeater.cc:347] Connected to a master server at 127.7.17.254:46765
I20260927 15:10:40.374612 23497 heartbeater.cc:464] Registering TS with master...
I20260927 15:10:40.375274 23497 heartbeater.cc:511] Master 127.7.17.254:46765 requested a full tablet report, sending...
I20260927 15:10:40.376561 23169 ts_manager.cc:194] Registered new tserver with Master: 9a560782e4d14bf2a19685db01492181 (127.7.17.194:39193)
I20260927 15:10:40.377391 23169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:33357
W20260927 15:10:40.484490 23500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:10:40.484789 23500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:10:40.484863 23500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:10:40.489095 23500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:10:40.489171 23500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:10:40.489233 23500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:10:40.494908 23500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:38749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=45207
--tserver_master_addrs=127.7.17.254:46765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:10:40.496085 23500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:10:40.497123 23500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:10:40.507022 23500 server_base.cc:1034] running on GCE node
W20260927 15:10:40.506996 23506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:40.506973 23508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:10:40.506973 23505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:10:40.507681 23500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:10:40.508332 23500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:10:40.509501 23500 hybrid_clock.cc:648] HybridClock initialized: now 1790521840509478 us; error 38 us; skew 500 ppm
Sep 27 15:10:40 dist-test-slave-hhpq krb5kdc[22592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521840, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:10:40.512185 23500 init.cc:377] Logged in from keytab as kudu/127.7.17.195@KRBTEST.COM (short username kudu)
I20260927 15:10:40.513156 23500 webserver.cc:460] Webserver started at http://127.7.17.195:45207/ using document root <none> and password file <none>
I20260927 15:10:40.513723 23500 fs_manager.cc:359] Metadata directory not provided
I20260927 15:10:40.513795 23500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:10:40.517779 23500 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:10:40.519907 23515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:40.521298 23500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:10:40.521427 23500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "6b9e6dcbb0844c53976e942ccdd36229"
format_stamp: "Formatted at 2026-09-27 15:10:19 on dist-test-slave-hhpq"
server_key: "8285cb926540b1ae4d2e3ff10ab9f668"
server_key_iv: "35a1cbd1e3b68f72d4a1b13c5f07cd8c"
server_key_version: "encryptionkey@0"
I20260927 15:10:40.522029 23500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:10:40.545805 23500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:10:40.549258 23500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:10:40.549453 23500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:10:40.550083 23500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:10:40.550972 23500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:10:40.551023 23500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:40.551079 23500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:10:40.551106 23500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:10:40.560024 23500 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:38749
I20260927 15:10:40.560065 23628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:38749 every 8 connection(s)
I20260927 15:10:40.561169 23500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:10:40.565922 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 23500
Sep 27 15:10:40 dist-test-slave-hhpq krb5kdc[22592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521840, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:10:40.575537 23633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:40.566646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51938 (local address 127.7.17.254:46765)
0927 15:10:40.566936 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:40.566940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:40.567080 (+ 140us) server_negotiation.cc:407] Connection header received
0927 15:10:40.567202 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:40.567204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:40.567241 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:40.567327 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:40.568165 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.568897 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.570161 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.570388 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.570486 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:40.572185 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:40.572188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:40.572190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:40.572215 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:40.573824 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:40.574312 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:40.574316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:40.574318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:40.574409 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:40.574687 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:40.574690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:40.574692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:40.574879 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:40.575096 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:40.575224 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 15:10:40.575343 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":103,"threads_started":1}
I20260927 15:10:40.578395 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:40.563216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:54941 (local address 127.7.17.254:46765)
0927 15:10:40.563409 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:40.563413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:40.564501 (+ 1088us) server_negotiation.cc:407] Connection header received
0927 15:10:40.566056 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:40.566061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:40.566130 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:40.566203 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:40.567931 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.568686 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.570417 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.570678 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.570751 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:40.573704 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:40.573708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:40.573711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:40.573753 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:40.575240 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:40.575907 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:40.575910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:40.575912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:40.575987 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:40.576539 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:40.576542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:40.576543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:40.576728 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:40.576813 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:40.578117 (+ 1304us) server_negotiation.cc:299] Negotiation successful
0927 15:10:40.578246 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 15:10:40.579075 23631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:40.563712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.195:54941)
0927 15:10:40.564264 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:40.564324 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:40.565698 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:40.566446 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:40.566458 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:40.566950 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:10:40.567711 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:40.567736 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.568898 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.568902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:40.570309 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:40.570320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.570869 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.570874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:40.571047 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:40.571551 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:10:40.571575 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:10:40.573465 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:10:40.575400 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:40.575407 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:40.575428 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:40.575803 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:40.576077 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:10:40.576080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:10:40.576081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:10:40.576291 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:10:40.576871 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:10:40.576878 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:10:40.577247 (+ 369us) client_negotiation.cc:769] Sending connection context
0927 15:10:40.577487 (+ 240us) client_negotiation.cc:240] Negotiation successful
0927 15:10:40.577900 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":126,"threads_started":1}
I20260927 15:10:40.580199 23629 heartbeater.cc:347] Connected to a master server at 127.7.17.254:46765
I20260927 15:10:40.580422 23629 heartbeater.cc:464] Registering TS with master...
I20260927 15:10:40.580986 23629 heartbeater.cc:511] Master 127.7.17.254:46765 requested a full tablet report, sending...
I20260927 15:10:40.581971 23169 ts_manager.cc:194] Registered new tserver with Master: 6b9e6dcbb0844c53976e942ccdd36229 (127.7.17.195:38749)
I20260927 15:10:40.582532 23169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:54941
I20260927 15:10:40.590287 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:10:40.594274 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:40.590889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51948 (local address 127.7.17.254:46765)
0927 15:10:40.591027 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:40.591032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:40.591237 (+ 205us) server_negotiation.cc:407] Connection header received
0927 15:10:40.591376 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:40.591378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:40.591485 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:40.591629 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:10:40.591794 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.592306 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.593258 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.593464 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.593566 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:40.593799 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:10:40.593943 (+ 144us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 15:10:40.594147 (+ 204us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51948: token expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 15:10:40.594362 23361 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51948: token expired
W20260927 15:10:40.594676 23101 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.7.17.254:46765)
I20260927 15:10:40.601406 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:40.594995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51958 (local address 127.7.17.254:46765)
0927 15:10:40.595108 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:40.595112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:40.595124 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:10:40.595185 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:40.595188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:40.595221 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:40.595278 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:10:40.595836 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.596299 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.597124 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.597323 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.597392 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:40.598604 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:10:40.598607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:10:40.598609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:10:40.598632 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:10:40.600006 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:40.600404 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:40.600406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:40.600408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:40.600478 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:10:40.600687 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:10:40.600689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:10:40.600690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:10:40.600899 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:10:40.600993 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:40.601165 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 15:10:40.601260 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:10:40.604110 23169 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:10:40.607435 23169 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:10:40.615512 23293 tablet_service.cc:1467] Processing CreateTablet for tablet f1bab22a24544b11a4dc0e0c5d12ba18 (DEFAULT_TABLE table=test-table [id=207cf7fb0ee141ac80963fdd19495036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:10:40.616214 23293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1bab22a24544b11a4dc0e0c5d12ba18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:10:40.619170 23251 tablet_bootstrap.cc:492] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21: Bootstrap starting.
I20260927 15:10:40.620014 23638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:40.614677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:39193 (local address 127.0.0.1:34008)
0927 15:10:40.615109 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:40.615122 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:40.615202 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:40.615933 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:40.615936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:40.615955 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:40.616072 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:40.616079 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.617329 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.617333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:40.618499 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:40.618510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.619638 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.619641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:40.619718 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:40.619758 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 15:10:40.619797 (+ 39us) client_negotiation.cc:240] Negotiation successful
0927 15:10:40.619855 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":152,"threads_started":1}
I20260927 15:10:40.620141 23251 tablet_bootstrap.cc:654] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21: Neither blocks nor log segments found. Creating new log.
I20260927 15:10:40.620309 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:40.614460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:38749 (local address 127.0.0.1:42138)
0927 15:10:40.614857 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:40.614874 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:40.614959 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:40.616063 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:40.616066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:40.616084 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:40.616196 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:40.616201 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.617927 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.617931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:40.618899 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:40.618907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.619937 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.619941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:40.620042 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:40.620064 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:10:40.620131 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 15:10:40.620211 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317}
I20260927 15:10:40.620438 23639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:40.614689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42138 (local address 127.7.17.195:38749)
0927 15:10:40.615594 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:40.615606 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:40.615629 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:10:40.615705 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:40.615710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:40.615910 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:40.616348 (+ 438us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:40.616529 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.617808 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.619009 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.619793 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.619916 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:40.620075 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:40.620184 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 15:10:40.620294 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":198,"threads_started":1}
I20260927 15:10:40.620993 23640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:40.615150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34008 (local address 127.7.17.194:39193)
0927 15:10:40.615432 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:40.615447 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:40.615472 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:10:40.615543 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:40.615548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:40.615734 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:40.615950 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:40.616165 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.617213 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.618637 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.619506 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.620527 (+ 1021us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:40.620687 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:40.620738 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 15:10:40.620855 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":82,"threads_started":1}
I20260927 15:10:40.621982 23251 tablet_bootstrap.cc:492] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21: No bootstrap required, opened a new log
I20260927 15:10:40.622094 23251 ts_tablet_manager.cc:1397] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 15:10:40.622539 23251 raft_consensus.cc:348] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } } peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } }
I20260927 15:10:40.622663 23251 raft_consensus.cc:374] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:10:40.622700 23251 raft_consensus.cc:729] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a1b0c9c4d49448f9d40a69effa00d21, State: Initialized, Role: FOLLOWER
I20260927 15:10:40.622823 23251 consensus_queue.cc:260] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } } peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } }
I20260927 15:10:40.623167 23251 ts_tablet_manager.cc:1428] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:10:40.623476 23563 tablet_service.cc:1467] Processing CreateTablet for tablet f1bab22a24544b11a4dc0e0c5d12ba18 (DEFAULT_TABLE table=test-table [id=207cf7fb0ee141ac80963fdd19495036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:10:40.623838 23431 tablet_service.cc:1467] Processing CreateTablet for tablet f1bab22a24544b11a4dc0e0c5d12ba18 (DEFAULT_TABLE table=test-table [id=207cf7fb0ee141ac80963fdd19495036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:10:40.624554 23563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1bab22a24544b11a4dc0e0c5d12ba18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:10:40.624960 23431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1bab22a24544b11a4dc0e0c5d12ba18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:10:40.631690 23642 tablet_bootstrap.cc:492] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181: Bootstrap starting.
I20260927 15:10:40.632336 23643 tablet_bootstrap.cc:492] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229: Bootstrap starting.
I20260927 15:10:40.634186 23643 tablet_bootstrap.cc:654] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229: Neither blocks nor log segments found. Creating new log.
I20260927 15:10:40.634641 23642 tablet_bootstrap.cc:654] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181: Neither blocks nor log segments found. Creating new log.
I20260927 15:10:40.635342 23643 log.cc:824] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:40.635456 23642 log.cc:824] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181: Log is configured to *not* fsync() on all Append() calls
I20260927 15:10:40.637609 23642 tablet_bootstrap.cc:492] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181: No bootstrap required, opened a new log
I20260927 15:10:40.637609 23643 tablet_bootstrap.cc:492] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229: No bootstrap required, opened a new log
I20260927 15:10:40.637827 23643 ts_tablet_manager.cc:1397] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:10:40.637827 23642 ts_tablet_manager.cc:1397] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:10:40.641930 23642 raft_consensus.cc:348] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } } peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } }
I20260927 15:10:40.641930 23643 raft_consensus.cc:348] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } } peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } }
I20260927 15:10:40.642261 23642 raft_consensus.cc:374] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:10:40.642261 23643 raft_consensus.cc:374] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:10:40.642321 23642 raft_consensus.cc:729] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a560782e4d14bf2a19685db01492181, State: Initialized, Role: FOLLOWER
I20260927 15:10:40.642321 23643 raft_consensus.cc:729] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9e6dcbb0844c53976e942ccdd36229, State: Initialized, Role: FOLLOWER
I20260927 15:10:40.642832 23642 consensus_queue.cc:260] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } } peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } }
I20260927 15:10:40.642832 23643 consensus_queue.cc:260] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } } peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } }
I20260927 15:10:40.643607 23629 heartbeater.cc:503] Master 127.7.17.254:46765 was elected leader, sending a full tablet report...
I20260927 15:10:40.643847 23497 heartbeater.cc:503] Master 127.7.17.254:46765 was elected leader, sending a full tablet report...
I20260927 15:10:40.643947 23642 ts_tablet_manager.cc:1428] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:10:40.643947 23643 ts_tablet_manager.cc:1428] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 15:10:40.657058 23360 tablet.cc:2369] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:10:40.751425 23646 raft_consensus.cc:482] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:10:40.751639 23646 raft_consensus.cc:504] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } } peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } }
I20260927 15:10:40.752770 23646 leader_election.cc:288] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b9e6dcbb0844c53976e942ccdd36229 (127.7.17.195:38749), 0a1b0c9c4d49448f9d40a69effa00d21 (127.7.17.193:32919)
I20260927 15:10:40.757898 23648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:40.753269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:44565 (local address 127.7.17.193:32919)
0927 15:10:40.753621 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:40.753625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:40.753641 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:10:40.753693 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:40.753695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:40.753750 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:40.753855 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:40.754170 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.754980 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.756620 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.757310 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.757478 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:40.757606 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:40.757669 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 15:10:40.757750 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":95,"threads_started":1}
I20260927 15:10:40.757907 23639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:40.753190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:53071 (local address 127.7.17.195:38749)
0927 15:10:40.753315 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:40.753321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:40.753839 (+ 518us) server_negotiation.cc:407] Connection header received
0927 15:10:40.753937 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:40.753940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:40.753989 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:40.754049 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:10:40.754290 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.755027 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.756620 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.757261 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.757398 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:40.757456 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:40.757677 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 15:10:40.757750 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:10:40.757920 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:40.753092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:32919 (local address 127.7.17.194:44565)
0927 15:10:40.753202 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:40.753217 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:40.753331 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:40.753923 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:40.753925 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:40.753941 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:40.754052 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:40.754059 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.755119 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.755123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:40.756465 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:40.756477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.757463 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.757466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:40.757568 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:40.757584 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 15:10:40.757646 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 15:10:40.757746 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":74}
I20260927 15:10:40.757927 23649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:40.753091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:38749 (local address 127.7.17.194:53071)
0927 15:10:40.753739 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:40.753751 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:10:40.753848 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:10:40.754081 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:10:40.754084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:10:40.754098 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:10:40.754194 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:40.754199 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.755127 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.755130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:40.756465 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:10:40.756477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.757427 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.757431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:10:40.757538 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:40.757568 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 15:10:40.757647 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 15:10:40.757746 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":362,"thread_start_us":119,"threads_started":1}
I20260927 15:10:40.758724 23313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1bab22a24544b11a4dc0e0c5d12ba18" candidate_uuid: "9a560782e4d14bf2a19685db01492181" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" is_pre_election: true
I20260927 15:10:40.758860 23583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1bab22a24544b11a4dc0e0c5d12ba18" candidate_uuid: "9a560782e4d14bf2a19685db01492181" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9e6dcbb0844c53976e942ccdd36229" is_pre_election: true
I20260927 15:10:40.759042 23313 raft_consensus.cc:2454] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a560782e4d14bf2a19685db01492181 in term 0.
I20260927 15:10:40.759183 23583 raft_consensus.cc:2454] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a560782e4d14bf2a19685db01492181 in term 0.
I20260927 15:10:40.759615 23385 leader_election.cc:302] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0a1b0c9c4d49448f9d40a69effa00d21, 6b9e6dcbb0844c53976e942ccdd36229, 9a560782e4d14bf2a19685db01492181; no voters:
I20260927 15:10:40.759908 23646 raft_consensus.cc:2781] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:10:40.760015 23646 raft_consensus.cc:482] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:10:40.760066 23646 raft_consensus.cc:3037] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:10:40.761220 23646 raft_consensus.cc:504] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } } peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } }
I20260927 15:10:40.761581 23646 leader_election.cc:288] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [CANDIDATE]: Term 1 election: Requested vote from peers 6b9e6dcbb0844c53976e942ccdd36229 (127.7.17.195:38749), 0a1b0c9c4d49448f9d40a69effa00d21 (127.7.17.193:32919)
I20260927 15:10:40.762006 23583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1bab22a24544b11a4dc0e0c5d12ba18" candidate_uuid: "9a560782e4d14bf2a19685db01492181" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9e6dcbb0844c53976e942ccdd36229"
I20260927 15:10:40.762032 23313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1bab22a24544b11a4dc0e0c5d12ba18" candidate_uuid: "9a560782e4d14bf2a19685db01492181" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a1b0c9c4d49448f9d40a69effa00d21"
I20260927 15:10:40.762156 23313 raft_consensus.cc:3037] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:10:40.762169 23583 raft_consensus.cc:3037] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:10:40.763398 23583 raft_consensus.cc:2454] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a560782e4d14bf2a19685db01492181 in term 1.
I20260927 15:10:40.763398 23313 raft_consensus.cc:2454] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a560782e4d14bf2a19685db01492181 in term 1.
I20260927 15:10:40.763720 23385 leader_election.cc:302] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a1b0c9c4d49448f9d40a69effa00d21, 9a560782e4d14bf2a19685db01492181; no voters:
I20260927 15:10:40.763907 23646 raft_consensus.cc:2781] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:10:40.764153 23646 raft_consensus.cc:686] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [term 1 LEADER]: Becoming Leader. State: Replica: 9a560782e4d14bf2a19685db01492181, State: Running, Role: LEADER
I20260927 15:10:40.764483 23646 consensus_queue.cc:237] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } } peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } }
I20260927 15:10:40.767838 23169 catalog_manager.cc:5495] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a560782e4d14bf2a19685db01492181 (127.7.17.194). New cstate: current_term: 1 leader_uuid: "9a560782e4d14bf2a19685db01492181" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } health_report { overall_health: UNKNOWN } } }
W20260927 15:10:40.812230 23630 tablet.cc:2369] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:10:40.815557 23640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:40.811931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34010 (local address 127.7.17.194:39193)
0927 15:10:40.812048 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 15:10:40.812051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:10:40.812065 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:10:40.812197 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:10:40.812199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:10:40.812257 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:10:40.812362 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:10:40.812688 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.813269 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.814137 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:10:40.814342 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:10:40.814424 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:10:40.814716 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:10:40.815186 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:10:40.815281 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:10:40.815308 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 15:10:40.815365 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:10:40.822093 23313 raft_consensus.cc:1261] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 1 FOLLOWER]: Refusing update from remote peer 9a560782e4d14bf2a19685db01492181: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:10:40.822093 23583 raft_consensus.cc:1261] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229 [term 1 FOLLOWER]: Refusing update from remote peer 9a560782e4d14bf2a19685db01492181: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:10:40.822882 23646 consensus_queue.cc:1035] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:10:40.823105 23650 consensus_queue.cc:1035] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:10:40.830163 23654 mvcc.cc:205] Tried to move back new op lower bound from 7333977459998609408 to 7333977459774083072. Current Snapshot: MvccSnapshot[applied={T|T < 7333977459998609408}]
I20260927 15:10:40.831324 23653 mvcc.cc:205] Tried to move back new op lower bound from 7333977459998609408 to 7333977459774083072. Current Snapshot: MvccSnapshot[applied={T|T < 7333977459998609408 or (T in {7333977459998609408})}]
I20260927 15:10:40.838213 23169 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:10:40.838421 23169 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:10:40.841033 23169 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b: Sending DeleteTablet for 3 replicas of tablet f1bab22a24544b11a4dc0e0c5d12ba18
I20260927 15:10:40.841832 23293 tablet_service.cc:1514] Processing DeleteTablet for tablet f1bab22a24544b11a4dc0e0c5d12ba18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:10:40 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:37800
I20260927 15:10:40.842145 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 23209
I20260927 15:10:40.842123 23431 tablet_service.cc:1514] Processing DeleteTablet for tablet f1bab22a24544b11a4dc0e0c5d12ba18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:10:40 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:34008
I20260927 15:10:40.842288 23563 tablet_service.cc:1514] Processing DeleteTablet for tablet f1bab22a24544b11a4dc0e0c5d12ba18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:10:40 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:42138
I20260927 15:10:40.845075 23663 tablet_replica.cc:330] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181: stopping tablet replica
I20260927 15:10:40.845106 23664 tablet_replica.cc:330] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229: stopping tablet replica
I20260927 15:10:40.845507 23663 raft_consensus.cc:2229] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:10:40.845508 23664 raft_consensus.cc:2229] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:10:40.845897 23664 raft_consensus.cc:2258] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:10:40.845999 23663 raft_consensus.cc:2258] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:10:40.847757 23664 ts_tablet_manager.cc:1905] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 15:10:40.848311 23156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.17.193:32919 (error 108)
I20260927 15:10:40.848493 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 23363
I20260927 15:10:40.850696 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:40.850382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:32919 (local address 127.0.0.1:37812)
0927 15:10:40.850520 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:40.850603 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.193:32919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 15:10:40.850880 23156 catalog_manager.cc:4584] TS 0a1b0c9c4d49448f9d40a69effa00d21 (127.7.17.193:32919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f1bab22a24544b11a4dc0e0c5d12ba18: Network error: Client connection negotiation failed: client connection to 127.7.17.193:32919: connect: Connection refused (error 111)
I20260927 15:10:40.851533 23664 ts_tablet_manager.cc:1918] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:10:40.851639 23664 log.cc:1196] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/f1bab22a24544b11a4dc0e0c5d12ba18
I20260927 15:10:40.851953 23664 ts_tablet_manager.cc:1939] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229: Deleting consensus metadata
I20260927 15:10:40.853304 23158 catalog_manager.cc:4841] TS 6b9e6dcbb0844c53976e942ccdd36229 (127.7.17.195:38749): tablet f1bab22a24544b11a4dc0e0c5d12ba18 (table test-table [id=207cf7fb0ee141ac80963fdd19495036]) successfully deleted
I20260927 15:10:40.855213 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 23500
I20260927 15:10:40.855406 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:10:40.855126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:39193 (local address 127.0.0.1:34024)
0927 15:10:40.855265 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 15:10:40.855324 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.194:39193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 15:10:40.860427 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 23138
I20260927 15:11:00.866519 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:46765
--webserver_interface=127.7.17.254
--webserver_port=36883
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:46765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:00.980041 23666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:00.980377 23666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:00.980438 23666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:00.984557 23666 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 15:11:00.984627 23666 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 15:11:00.984648 23666 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 15:11:00.984665 23666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:00.984689 23666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:00.984721 23666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:00.984751 23666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:00.984781 23666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:00.990242 23666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:46765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:46765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=36883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:00.991361 23666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:00.992456 23666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:01.001274 23671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:01.001422 23666 server_base.cc:1034] running on GCE node
W20260927 15:11:01.001248 23672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:01.001335 23674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:01.002023 23666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:01.003166 23666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:01.004341 23666 hybrid_clock.cc:648] HybridClock initialized: now 1790521861004311 us; error 44 us; skew 500 ppm
Sep 27 15:11:01 dist-test-slave-hhpq krb5kdc[22592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521861, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:01.007123 23666 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:11:01.008108 23666 webserver.cc:460] Webserver started at http://127.7.17.254:36883/ using document root <none> and password file <none>
I20260927 15:11:01.008652 23666 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:01.008713 23666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:01.012753 23666 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:01.015210 23681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:01.016649 23666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 15:11:01.016799 23666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "af580a3e658c4e8a9731da421183718b"
format_stamp: "Formatted at 2026-09-27 15:10:18 on dist-test-slave-hhpq"
server_key: "1364626bf50f3700bf560ab9017792db"
server_key_iv: "b4bda720f322c0eb339345d71e7d85fe"
server_key_version: "encryptionkey@0"
I20260927 15:11:01.017445 23666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:01.030715 23666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:01.034046 23666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:01.034246 23666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:01.041602 23666 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:46765
I20260927 15:11:01.041862 23733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:46765 every 8 connection(s)
I20260927 15:11:01.042752 23666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:01.043098 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 23666
I20260927 15:11:01.043519 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:32919
--local_ip_for_outbound_sockets=127.7.17.193
--tserver_master_addrs=127.7.17.254:46765
--webserver_port=36101
--webserver_interface=127.7.17.193
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:11:01.047848 23734 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:11:01.050416 23734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b: Bootstrap starting.
I20260927 15:11:01.073952 23734 log.cc:824] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:01.088300 23734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:11:01.088801 23734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b: Bootstrap complete.
I20260927 15:11:01.094302 23734 raft_consensus.cc:348] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af580a3e658c4e8a9731da421183718b" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46765 } }
I20260927 15:11:01.094874 23734 raft_consensus.cc:729] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: af580a3e658c4e8a9731da421183718b, State: Initialized, Role: FOLLOWER
I20260927 15:11:01.095472 23734 consensus_queue.cc:260] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af580a3e658c4e8a9731da421183718b" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46765 } }
I20260927 15:11:01.095633 23734 raft_consensus.cc:388] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:01.095721 23734 raft_consensus.cc:482] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:01.095821 23734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 2 FOLLOWER]: Advancing to term 3
I20260927 15:11:01.098497 23734 raft_consensus.cc:504] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af580a3e658c4e8a9731da421183718b" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46765 } }
I20260927 15:11:01.098928 23734 leader_election.cc:302] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af580a3e658c4e8a9731da421183718b; no voters:
I20260927 15:11:01.099308 23734 leader_election.cc:288] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 15:11:01.099512 23738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 3 FOLLOWER]: Leader election won for term 3
I20260927 15:11:01.099757 23738 raft_consensus.cc:686] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [term 3 LEADER]: Becoming Leader. State: Replica: af580a3e658c4e8a9731da421183718b, State: Running, Role: LEADER
I20260927 15:11:01.100126 23738 consensus_queue.cc:237] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af580a3e658c4e8a9731da421183718b" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46765 } }
I20260927 15:11:01.100692 23734 sys_catalog.cc:564] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:01.101459 23739 sys_catalog.cc:455] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "af580a3e658c4e8a9731da421183718b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af580a3e658c4e8a9731da421183718b" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46765 } } }
I20260927 15:11:01.101578 23739 sys_catalog.cc:458] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:01.102223 23740 sys_catalog.cc:455] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [sys.catalog]: SysCatalogTable state changed. Reason: New leader af580a3e658c4e8a9731da421183718b. Latest consensus state: current_term: 3 leader_uuid: "af580a3e658c4e8a9731da421183718b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af580a3e658c4e8a9731da421183718b" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46765 } } }
I20260927 15:11:01.102326 23740 sys_catalog.cc:458] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:01.102355 23746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:01.107594 23746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:01.108394 23746 catalog_manager.cc:1259] Loaded cluster ID: 936dd6b417434a0c971f8674f81c4d4a
I20260927 15:11:01.108462 23746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:01.111009 23746 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:01.111845 23746 catalog_manager.cc:5879] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b: Loaded TSK: 0
I20260927 15:11:01.111934 23746 catalog_manager.cc:5879] T 00000000000000000000000000000000 P af580a3e658c4e8a9731da421183718b: Loaded TSK: 1
F20260927 15:11:01.113078 23746 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f85530a19ad google::LogMessage::Fail() at ??:0
@ 0x7f85530a488a google::LogMessage::SendToLog() at ??:0
@ 0x7f85530a14de google::LogMessage::Flush() at ??:0
@ 0x7f85530a2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8555d6518d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f8555d5df0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f855da067bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f855d9e099c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f855d9e1178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f855da22df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f855da4182b std::function<>::operator()() at ??:0
@ 0x7f855d9e0b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f855d9e1f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f855d9de579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f855da22640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f85552f33d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f85552f3c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f85552f53f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f85552e4512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f85569286db start_thread at ??:0
@ 0x7f8551b9f61f clone at ??:0
W20260927 15:11:01.207832 23736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:01.208113 23736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:01.208158 23736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:01.212399 23736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:01.212471 23736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:01.212536 23736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:11:01.218165 23736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:32919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=36101
--tserver_master_addrs=127.7.17.254:46765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:01.219300 23736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:01.220325 23736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:01.230084 23783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:01.230103 23782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:01.230248 23736 server_base.cc:1034] running on GCE node
W20260927 15:11:01.230508 23785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:01.231031 23736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:01.231953 23736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:01.233153 23736 hybrid_clock.cc:648] HybridClock initialized: now 1790521861233116 us; error 47 us; skew 500 ppm
Sep 27 15:11:01 dist-test-slave-hhpq krb5kdc[22592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521861, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:01.236555 23736 init.cc:377] Logged in from keytab as kudu/127.7.17.193@KRBTEST.COM (short username kudu)
I20260927 15:11:01.237566 23736 webserver.cc:460] Webserver started at http://127.7.17.193:36101/ using document root <none> and password file <none>
I20260927 15:11:01.238340 23736 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:01.238435 23736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:01.242786 23736 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 15:11:01.247007 23792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:01.248879 23736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:11:01.249037 23736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "0a1b0c9c4d49448f9d40a69effa00d21"
format_stamp: "Formatted at 2026-09-27 15:10:18 on dist-test-slave-hhpq"
server_key: "47927d8866c8b84da96eec2b19045bf3"
server_key_iv: "d7b72d30db6d4f10bde001e5f95ae503"
server_key_version: "encryptionkey@0"
I20260927 15:11:01.249636 23736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:01.284777 23736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:01.288143 23736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:01.288343 23736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:01.288992 23736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:01.290143 23799 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 15:11:01.295976 23736 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 15:11:01.296082 23736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 15:11:01.296161 23736 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 15:11:01.298066 23736 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 15:11:01.298130 23736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260927 15:11:01.298477 23799 tablet_bootstrap.cc:492] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21: Bootstrap starting.
I20260927 15:11:01.308326 23736 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:32919
I20260927 15:11:01.308604 23906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:32919 every 8 connection(s)
I20260927 15:11:01.309479 23736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:11:01.311659 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 23736
I20260927 15:11:01.312148 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:39193
--local_ip_for_outbound_sockets=127.7.17.194
--tserver_master_addrs=127.7.17.254:46765
--webserver_port=45353
--webserver_interface=127.7.17.194
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:11:01.323280 23799 log.cc:824] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:01.332792 23799 tablet_bootstrap.cc:492] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:11:01.334194 23799 tablet_bootstrap.cc:492] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21: Bootstrap complete.
I20260927 15:11:01.335512 23799 ts_tablet_manager.cc:1397] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21: Time spent bootstrapping tablet: real 0.037s user 0.021s sys 0.011s
I20260927 15:11:01.338881 23799 raft_consensus.cc:348] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } } peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } }
I20260927 15:11:01.339422 23799 raft_consensus.cc:729] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a1b0c9c4d49448f9d40a69effa00d21, State: Initialized, Role: FOLLOWER
I20260927 15:11:01.339921 23799 consensus_queue.cc:260] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } } peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } }
I20260927 15:11:01.340920 23799 ts_tablet_manager.cc:1428] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
W20260927 15:11:01.341063 23909 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:01.311599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.193:37293)
0927 15:11:01.313265 (+ 1666us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:01.313336 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:01.314455 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:01.340088 (+ 25633us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1385,"thread_start_us":154,"threads_started":1}
W20260927 15:11:01.341536 23907 heartbeater.cc:644] Failed to heartbeat to 127.7.17.254:46765 (0 consecutive failures): Network error: Failed to ping master at 127.7.17.254:46765: Client connection negotiation failed: client connection to 127.7.17.254:46765: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 15:11:01.342242 23909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:01.341959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.193:50419)
0927 15:11:01.342086 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:01.342163 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 15:11:01.342935 23909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:01.342720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.193:55251)
0927 15:11:01.342827 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:01.342878 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 15:11:01.343096 23907 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:11:01.432772 23910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:01.433122 23910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:01.433168 23910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:01.437301 23910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:01.437382 23910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:01.437445 23910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:11:01.443063 23910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:39193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=45353
--tserver_master_addrs=127.7.17.254:46765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:01.444159 23910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:01.445225 23910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:01.455433 23920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:01.455497 23918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:01.455433 23917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:01.455647 23910 server_base.cc:1034] running on GCE node
I20260927 15:11:01.456193 23910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:01.456934 23910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:01.458097 23910 hybrid_clock.cc:648] HybridClock initialized: now 1790521861458075 us; error 37 us; skew 500 ppm
Sep 27 15:11:01 dist-test-slave-hhpq krb5kdc[22592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521861, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:01.461172 23910 init.cc:377] Logged in from keytab as kudu/127.7.17.194@KRBTEST.COM (short username kudu)
I20260927 15:11:01.462194 23910 webserver.cc:460] Webserver started at http://127.7.17.194:45353/ using document root <none> and password file <none>
I20260927 15:11:01.462782 23910 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:01.462849 23910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:01.467063 23910 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:11:01.469471 23927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:01.470949 23910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:01.471091 23910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "9a560782e4d14bf2a19685db01492181"
format_stamp: "Formatted at 2026-09-27 15:10:19 on dist-test-slave-hhpq"
server_key: "6436e3aa564b581374fa03abac4e10bc"
server_key_iv: "58fe4fcb26abdbb22d3e8fc8aa9e4f24"
server_key_version: "encryptionkey@0"
I20260927 15:11:01.471719 23910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:01.490967 23910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:01.494632 23910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:01.494828 23910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:01.495467 23910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:01.496652 23934 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 15:11:01.501019 23910 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 15:11:01.501106 23910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 15:11:01.501183 23910 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 15:11:01.503014 23910 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 15:11:01.503075 23910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 15:11:01.503432 23934 tablet_bootstrap.cc:492] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181: Bootstrap starting.
I20260927 15:11:01.512650 23910 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:39193
I20260927 15:11:01.512794 24041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:39193 every 8 connection(s)
I20260927 15:11:01.513775 23910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:11:01.517947 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:01.516019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:38401)
0927 15:11:01.516647 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:01.516870 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":167,"threads_started":1}
W20260927 15:11:01.518437 24042 heartbeater.cc:644] Failed to heartbeat to 127.7.17.254:46765 (0 consecutive failures): Network error: Failed to ping master at 127.7.17.254:46765: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
I20260927 15:11:01.519073 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:01.518813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:36325)
0927 15:11:01.518941 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:01.519004 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 15:11:01.519834 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:01.519567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:33293)
0927 15:11:01.519690 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:01.519752 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 15:11:01.520020 24042 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 15:11:01.521561 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 23910
I20260927 15:11:01.522038 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:38749
--local_ip_for_outbound_sockets=127.7.17.195
--tserver_master_addrs=127.7.17.254:46765
--webserver_port=45207
--webserver_interface=127.7.17.195
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:11:01.523089 23934 log.cc:824] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:01.532905 23934 tablet_bootstrap.cc:492] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:11:01.533401 23934 tablet_bootstrap.cc:492] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181: Bootstrap complete.
I20260927 15:11:01.534855 23934 ts_tablet_manager.cc:1397] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.004s
I20260927 15:11:01.538753 23934 raft_consensus.cc:348] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } } peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } }
I20260927 15:11:01.539392 23934 raft_consensus.cc:729] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a560782e4d14bf2a19685db01492181, State: Initialized, Role: FOLLOWER
I20260927 15:11:01.539956 23934 consensus_queue.cc:260] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } } peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } }
I20260927 15:11:01.541186 23934 ts_tablet_manager.cc:1428] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260927 15:11:01.541313 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:01.541025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:38855)
0927 15:11:01.541157 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:01.541223 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 15:11:01.542079 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:01.541790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:55163)
0927 15:11:01.541940 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:01.542003 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 15:11:01.640625 24045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:01.640952 24045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:01.641001 24045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:01.645198 24045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:01.645272 24045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:01.645336 24045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:11:01.651170 24045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:38749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=45207
--tserver_master_addrs=127.7.17.254:46765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:01.652313 24045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:01.653439 24045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:01.663331 24055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:01.663406 24052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:01.663439 24053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:01.663666 24045 server_base.cc:1034] running on GCE node
I20260927 15:11:01.664139 24045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:01.664894 24045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:01.666072 24045 hybrid_clock.cc:648] HybridClock initialized: now 1790521861666041 us; error 51 us; skew 500 ppm
Sep 27 15:11:01 dist-test-slave-hhpq krb5kdc[22592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521861, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:01.668921 24045 init.cc:377] Logged in from keytab as kudu/127.7.17.195@KRBTEST.COM (short username kudu)
I20260927 15:11:01.669876 24045 webserver.cc:460] Webserver started at http://127.7.17.195:45207/ using document root <none> and password file <none>
I20260927 15:11:01.670450 24045 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:01.670521 24045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:01.674738 24045 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:11:01.677261 24062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:01.678699 24045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:01.678846 24045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "6b9e6dcbb0844c53976e942ccdd36229"
format_stamp: "Formatted at 2026-09-27 15:10:19 on dist-test-slave-hhpq"
server_key: "8285cb926540b1ae4d2e3ff10ab9f668"
server_key_iv: "35a1cbd1e3b68f72d4a1b13c5f07cd8c"
server_key_version: "encryptionkey@0"
I20260927 15:11:01.679437 24045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:01.699115 24045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:01.702564 24045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:01.702764 24045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:01.703393 24045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:01.704243 24045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:01.704290 24045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:01.704344 24045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:01.704372 24045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:01.713400 24045 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:38749
I20260927 15:11:01.713475 24175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:38749 every 8 connection(s)
I20260927 15:11:01.714511 24045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:11:01.718734 24178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:01.716981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.195:52741)
0927 15:11:01.717742 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:01.717870 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":136,"threads_started":1}
W20260927 15:11:01.719357 24176 heartbeater.cc:644] Failed to heartbeat to 127.7.17.254:46765 (0 consecutive failures): Network error: Failed to ping master at 127.7.17.254:46765: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
I20260927 15:11:01.719748 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 24045
I20260927 15:11:01.720099 24178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:01.719849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.195:40409)
0927 15:11:01.719969 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:01.720025 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 15:11:01.720717 24178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:01.720497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.195:39575)
0927 15:11:01.720614 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:01.720656 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 15:11:01.720973 24176 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 15:11:02.344166 24180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:02.343729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.193:52361)
0927 15:11:02.343987 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:02.344050 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
I20260927 15:11:02.543434 24181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:02.542939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:58327)
0927 15:11:02.543248 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:02.543317 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":77,"threads_started":1}
I20260927 15:11:02.722011 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:02.721580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.195:42889)
0927 15:11:02.721844 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:02.721904 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
I20260927 15:11:03.015512 24183 raft_consensus.cc:482] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:11:03.015704 24183 raft_consensus.cc:504] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } } peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } }
I20260927 15:11:03.016788 24183 leader_election.cc:288] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b9e6dcbb0844c53976e942ccdd36229 (127.7.17.195:38749), 0a1b0c9c4d49448f9d40a69effa00d21 (127.7.17.193:32919)
Sep 27 15:11:03 dist-test-slave-hhpq krb5kdc[22592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521861, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for kudu/127.7.17.193@KRBTEST.COM
Sep 27 15:11:03 dist-test-slave-hhpq krb5kdc[22592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521861, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for kudu/127.7.17.195@KRBTEST.COM
I20260927 15:11:03.031953 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:03.017014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:38749 (local address 127.7.17.194:38639)
0927 15:11:03.017378 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:03.017411 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:03.017914 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:03.019682 (+ 1768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:03.019685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:03.019855 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:03.020006 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:03.020026 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:03.021914 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:03.021918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:03.023556 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:03.023568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:03.024551 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:03.024555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:03.024821 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:03.025686 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:03.025703 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:03.027584 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:03.029572 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:03.029575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:03.029576 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:03.029886 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:03.030219 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:03.030222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:03.030224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:03.030365 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:03.030983 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:03.030989 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:03.031229 (+ 240us) client_negotiation.cc:769] Sending connection context
0927 15:11:03.031400 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 15:11:03.031767 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":291,"spinlock_wait_cycles":68608}
I20260927 15:11:03.031953 24186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:03.017373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:44963 (local address 127.7.17.193:32919)
0927 15:11:03.017640 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:03.017648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:03.017679 (+ 31us) server_negotiation.cc:407] Connection header received
0927 15:11:03.018337 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:03.018344 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:03.018646 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:03.018859 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:03.020199 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:03.021761 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:03.023684 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:03.024428 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:03.024594 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:03.027519 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:03.027528 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:03.027548 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:03.027584 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:03.029378 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:03.029996 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:03.030001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:03.030006 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:03.030128 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:03.030463 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:03.030466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:03.030467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:03.030870 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:03.030988 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:03.031528 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 15:11:03.031772 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":83,"threads_started":1}
I20260927 15:11:03.031970 24185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:03.017205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:38639 (local address 127.7.17.195:38749)
0927 15:11:03.017766 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:03.017786 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:03.017818 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:11:03.019034 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:03.019043 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:03.019444 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:03.019683 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:03.020183 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:03.021761 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:03.023684 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:03.024428 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:03.024594 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:03.027777 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:03.027783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:03.027801 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:03.027832 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:03.029461 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:03.029996 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:03.030001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:03.030005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:03.030123 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:03.030451 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:03.030454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:03.030456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:03.030872 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:03.030988 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:03.031531 (+ 543us) server_negotiation.cc:299] Negotiation successful
0927 15:11:03.031774 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":112,"threads_started":1}
I20260927 15:11:03.032852 24181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:03.017070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:32919 (local address 127.7.17.194:44963)
0927 15:11:03.017554 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:03.017566 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:03.017914 (+ 348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:03.018978 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:03.018988 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:03.019394 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:03.020006 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:03.020026 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:03.021914 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:03.021918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:03.023555 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:03.023568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:03.024551 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:03.024555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:03.024849 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:03.025634 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:03.025657 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:03.027259 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:03.029503 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:03.029510 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:03.029525 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:03.029883 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:03.030223 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:03.030226 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:03.030228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:03.030374 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:03.030983 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:03.030989 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:03.031228 (+ 239us) client_negotiation.cc:769] Sending connection context
0927 15:11:03.031400 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 15:11:03.031842 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":350,"spinlock_wait_cycles":12416,"thread_start_us":85,"threads_started":1}
I20260927 15:11:03.033991 23861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1bab22a24544b11a4dc0e0c5d12ba18" candidate_uuid: "9a560782e4d14bf2a19685db01492181" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" is_pre_election: true
I20260927 15:11:03.034351 23861 raft_consensus.cc:2454] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a560782e4d14bf2a19685db01492181 in term 1.
W20260927 15:11:03.034981 23931 leader_election.cc:341] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b9e6dcbb0844c53976e942ccdd36229 (127.7.17.195:38749): Not found: Tablet not found: f1bab22a24544b11a4dc0e0c5d12ba18
I20260927 15:11:03.035140 23931 leader_election.cc:302] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a1b0c9c4d49448f9d40a69effa00d21, 9a560782e4d14bf2a19685db01492181; no voters: 6b9e6dcbb0844c53976e942ccdd36229
I20260927 15:11:03.035416 24183 raft_consensus.cc:2781] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 15:11:03.035490 24183 raft_consensus.cc:482] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:11:03.035521 24183 raft_consensus.cc:3037] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:11:03.037875 24183 raft_consensus.cc:504] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } } peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } }
I20260927 15:11:03.038240 24183 leader_election.cc:288] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [CANDIDATE]: Term 2 election: Requested vote from peers 6b9e6dcbb0844c53976e942ccdd36229 (127.7.17.195:38749), 0a1b0c9c4d49448f9d40a69effa00d21 (127.7.17.193:32919)
I20260927 15:11:03.038640 23861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1bab22a24544b11a4dc0e0c5d12ba18" candidate_uuid: "9a560782e4d14bf2a19685db01492181" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0a1b0c9c4d49448f9d40a69effa00d21"
I20260927 15:11:03.038785 23861 raft_consensus.cc:3037] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 1 FOLLOWER]: Advancing to term 2
W20260927 15:11:03.038944 23931 leader_election.cc:341] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 6b9e6dcbb0844c53976e942ccdd36229 (127.7.17.195:38749): Not found: Tablet not found: f1bab22a24544b11a4dc0e0c5d12ba18
I20260927 15:11:03.041018 23861 raft_consensus.cc:2454] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a560782e4d14bf2a19685db01492181 in term 2.
I20260927 15:11:03.041335 23929 leader_election.cc:302] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a1b0c9c4d49448f9d40a69effa00d21, 9a560782e4d14bf2a19685db01492181; no voters: 6b9e6dcbb0844c53976e942ccdd36229
I20260927 15:11:03.041504 24183 raft_consensus.cc:2781] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:11:03.041687 24183 raft_consensus.cc:686] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [term 2 LEADER]: Becoming Leader. State: Replica: 9a560782e4d14bf2a19685db01492181, State: Running, Role: LEADER
I20260927 15:11:03.042027 24183 consensus_queue.cc:237] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } } peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } }
I20260927 15:11:03.042433 24181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:03.042159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:32933)
0927 15:11:03.042289 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:03.042354 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 15:11:03.345585 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:03.345069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.193:39461)
0927 15:11:03.345389 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:03.345464 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":103,"threads_started":1}
W20260927 15:11:03.429868 23931 consensus_peers.cc:487] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 -> Peer 6b9e6dcbb0844c53976e942ccdd36229 (127.7.17.195:38749): Couldn't send request to peer 6b9e6dcbb0844c53976e942ccdd36229. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1bab22a24544b11a4dc0e0c5d12ba18. This is attempt 1: this message will repeat every 5th retry.
I20260927 15:11:03.576184 23861 raft_consensus.cc:1261] T f1bab22a24544b11a4dc0e0c5d12ba18 P 0a1b0c9c4d49448f9d40a69effa00d21 [term 2 FOLLOWER]: Refusing update from remote peer 9a560782e4d14bf2a19685db01492181: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260927 15:11:03.577059 24183 consensus_queue.cc:1035] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 15:11:03.579175 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:03.578907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.193:46861)
0927 15:11:03.579042 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:03.579099 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 15:11:03.581960 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:03.581526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:43677)
0927 15:11:03.581806 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:03.581870 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":70,"threads_started":1}
I20260927 15:11:03.582203 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:03.581965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.193:34069)
0927 15:11:03.582080 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:03.582133 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 15:11:03.723158 24196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:03.722732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.195:55023)
0927 15:11:03.722986 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:03.723046 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260927 15:11:04.032162 24197 ts_tablet_manager.cc:927] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229: Initiating tablet copy from peer 9a560782e4d14bf2a19685db01492181 (127.7.17.194:39193)
I20260927 15:11:04.032809 24197 tablet_copy_client.cc:316] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229: tablet copy: Beginning tablet copy session from remote peer at address 127.7.17.194:39193
Sep 27 15:11:04 dist-test-slave-hhpq krb5kdc[22592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521861, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for kudu/127.7.17.194@KRBTEST.COM
I20260927 15:11:04.046967 24196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:04.033365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:39193 (local address 127.7.17.195:36381)
0927 15:11:04.033494 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:04.033515 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:04.033638 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:04.034260 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:04.034265 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:04.034668 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:04.034786 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:04.034792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:04.035821 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:04.035825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:04.036507 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:04.036517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:04.037221 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:04.037224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:04.037349 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:04.041273 (+ 3924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:04.041287 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:04.042915 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:04.044984 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:04.044994 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:04.044999 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:04.045297 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:04.045629 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:04.045631 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:04.045632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:04.045713 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:04.046395 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:04.046401 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:04.046528 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 15:11:04.046659 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 15:11:04.046820 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 15:11:04.047138 24199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:04.033540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:36381 (local address 127.7.17.194:39193)
0927 15:11:04.033840 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:04.033851 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:04.033874 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:11:04.033922 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:04.033926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:04.034110 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:04.034230 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:04.034924 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:04.035707 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:04.036632 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:04.037112 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:04.037206 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:04.043095 (+ 5889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:04.043108 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:04.043122 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:04.043151 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:04.044813 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:04.045439 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:04.045445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:04.045447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:04.045545 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:04.045832 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:04.045835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:04.045836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:04.046258 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:04.046387 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:04.046831 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 15:11:04.046977 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":99,"threads_started":1}
I20260927 15:11:04.048291 24016 tablet_copy_service.cc:140] P 9a560782e4d14bf2a19685db01492181: Received BeginTabletCopySession request for tablet f1bab22a24544b11a4dc0e0c5d12ba18 from peer 6b9e6dcbb0844c53976e942ccdd36229 ({username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:36381)
I20260927 15:11:04.048470 24016 tablet_copy_service.cc:161] P 9a560782e4d14bf2a19685db01492181: Beginning new tablet copy session on tablet f1bab22a24544b11a4dc0e0c5d12ba18 from peer 6b9e6dcbb0844c53976e942ccdd36229 at {username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:36381: session id = 6b9e6dcbb0844c53976e942ccdd36229-f1bab22a24544b11a4dc0e0c5d12ba18
I20260927 15:11:04.049697 24016 tablet_copy_source_session.cc:205] T f1bab22a24544b11a4dc0e0c5d12ba18 P 9a560782e4d14bf2a19685db01492181: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 15:11:04.054469 24197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1bab22a24544b11a4dc0e0c5d12ba18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:04.061488 24197 tablet_copy_client.cc:791] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229: tablet copy: Starting download of 0 data blocks...
I20260927 15:11:04.061698 24197 tablet_copy_client.cc:655] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229: tablet copy: Starting download of 1 WAL segments...
I20260927 15:11:04.063922 24197 tablet_copy_client.cc:531] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 15:11:04.065932 24197 tablet_bootstrap.cc:492] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229: Bootstrap starting.
I20260927 15:11:04.071842 24197 log.cc:824] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:04.079214 24197 tablet_bootstrap.cc:492] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:11:04.079530 24197 tablet_bootstrap.cc:492] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229: Bootstrap complete.
I20260927 15:11:04.079862 24197 ts_tablet_manager.cc:1397] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.003s
I20260927 15:11:04.082454 24197 raft_consensus.cc:348] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } } peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } }
I20260927 15:11:04.082769 24197 raft_consensus.cc:729] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9e6dcbb0844c53976e942ccdd36229, State: Initialized, Role: FOLLOWER
I20260927 15:11:04.083220 24197 consensus_queue.cc:260] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9e6dcbb0844c53976e942ccdd36229" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38749 } } peers { permanent_uuid: "9a560782e4d14bf2a19685db01492181" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 39193 } } peers { permanent_uuid: "0a1b0c9c4d49448f9d40a69effa00d21" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 32919 } }
I20260927 15:11:04.084185 24197 ts_tablet_manager.cc:1428] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260927 15:11:04.084465 24196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:04.084191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.195:35101)
0927 15:11:04.084317 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:04.084383 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 15:11:04.084978 24016 tablet_copy_service.cc:342] P 9a560782e4d14bf2a19685db01492181: Request end of tablet copy session 6b9e6dcbb0844c53976e942ccdd36229-f1bab22a24544b11a4dc0e0c5d12ba18 received from {username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:36381
I20260927 15:11:04.085103 24016 tablet_copy_service.cc:433] P 9a560782e4d14bf2a19685db01492181: ending tablet copy session 6b9e6dcbb0844c53976e942ccdd36229-f1bab22a24544b11a4dc0e0c5d12ba18 on tablet f1bab22a24544b11a4dc0e0c5d12ba18 with peer 6b9e6dcbb0844c53976e942ccdd36229
I20260927 15:11:04.085155 24196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:04.084930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.195:34749)
0927 15:11:04.085032 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:04.085082 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 15:11:04.583103 24202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:04.582670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:60035)
0927 15:11:04.582930 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:04.582992 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":92,"threads_started":1}
I20260927 15:11:04.583220 24203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:04.582799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.193:52205)
0927 15:11:04.583026 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:04.583075 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":69,"threads_started":1}
I20260927 15:11:04.643931 24202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:04.643634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:40117)
0927 15:11:04.643782 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:04.643843 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 15:11:04.644037 24130 raft_consensus.cc:1203] T f1bab22a24544b11a4dc0e0c5d12ba18 P 6b9e6dcbb0844c53976e942ccdd36229 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260927 15:11:04.645318 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:04.644914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.195:60157)
0927 15:11:04.645142 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:04.645210 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":78,"threads_started":1}
I20260927 15:11:05.584478 24208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:05.584015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.193:43605)
0927 15:11:05.584307 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:05.584375 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
I20260927 15:11:05.645264 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:05.644752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:35087)
0927 15:11:05.645059 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:05.645120 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":122,"threads_started":1}
I20260927 15:11:05.646392 24210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:05.645969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.195:48057)
0927 15:11:05.646249 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:05.646305 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":67,"threads_started":1}
I20260927 15:11:06.585913 24211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:06.585438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.193:39271)
0927 15:11:06.585730 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:06.585814 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":80,"threads_started":1}
I20260927 15:11:06.646729 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:06.646224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:59427)
0927 15:11:06.646515 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:06.646584 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":78,"threads_started":1}
I20260927 15:11:06.647647 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:06.647143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.195:58041)
0927 15:11:06.647489 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:06.647548 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":128,"threads_started":1}
I20260927 15:11:06.647864 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:06.647233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:57685)
0927 15:11:06.647654 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:06.647712 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 15:11:07.587046 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:07.586578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.193:43579)
0927 15:11:07.586889 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:07.586948 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":72,"threads_started":1}
I20260927 15:11:07.649102 24215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:07.648632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.195:56329)
0927 15:11:07.648934 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:07.649001 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":75,"threads_started":1}
I20260927 15:11:07.649190 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:07.648633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:39573)
0927 15:11:07.649041 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:07.649095 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":105,"threads_started":1}
I20260927 15:11:08.588430 24217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:08.587928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.193:43863)
0927 15:11:08.588261 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:08.588330 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":93,"threads_started":1}
I20260927 15:11:08.650362 24219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:08.649946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:32815)
0927 15:11:08.650211 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:08.650269 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":91,"threads_started":1}
I20260927 15:11:08.650362 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:08.649886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.195:39871)
0927 15:11:08.650190 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:08.650260 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":93,"threads_started":1}
I20260927 15:11:09.589761 24220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:09.589310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.193:58829)
0927 15:11:09.589596 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:09.589660 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":79,"threads_started":1}
I20260927 15:11:09.651648 24222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:09.651179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:60709)
0927 15:11:09.651481 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:09.651542 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":84,"threads_started":1}
I20260927 15:11:09.651682 24221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:09.651179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.195:40651)
0927 15:11:09.651529 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:09.651590 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":80,"threads_started":1}
I20260927 15:11:10.591104 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:10.590610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.193:57283)
0927 15:11:10.590924 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:10.591004 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":96,"threads_started":1}
I20260927 15:11:10.653028 24224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:10.652502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:54943)
0927 15:11:10.652866 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:10.652930 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":102,"threads_started":1}
I20260927 15:11:10.653030 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:10.652601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.195:33191)
0927 15:11:10.652865 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:10.652931 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":75,"threads_started":1}
I20260927 15:11:11.592612 24226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:11.592075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.193:48835)
0927 15:11:11.592409 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:11.592510 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":84,"threads_started":1}
I20260927 15:11:11.654412 24227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:11.653887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:40619)
0927 15:11:11.654260 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:11.654315 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":83,"threads_started":1}
I20260927 15:11:11.654413 24228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:11.653874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.195:50317)
0927 15:11:11.654252 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:11.654323 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":87,"threads_started":1}
I20260927 15:11:12.593943 24230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:12.593469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.193:40697)
0927 15:11:12.593751 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:12.593840 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":76,"threads_started":1}
I20260927 15:11:12.655746 24232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:12.655296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.195:46877)
0927 15:11:12.655567 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:12.655642 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":85,"threads_started":1}
I20260927 15:11:12.655771 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:12.655296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:50193)
0927 15:11:12.655632 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:12.655684 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":87,"threads_started":1}
I20260927 15:11:13.595345 24234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:13.594849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.193:54293)
0927 15:11:13.595159 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:13.595231 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":82,"threads_started":1}
I20260927 15:11:13.657140 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:13.656614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.195:58715)
0927 15:11:13.656969 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:13.657036 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":128,"threads_started":1}
I20260927 15:11:13.657172 24236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:13.656614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:36061)
0927 15:11:13.657030 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:13.657084 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":130,"threads_started":1}
I20260927 15:11:14.596598 24237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:14.596184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.193:54111)
0927 15:11:14.596436 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:14.596499 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
I20260927 15:11:14.658640 24239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:14.658059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.195:57683)
0927 15:11:14.658423 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:14.658517 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":83,"threads_started":1}
I20260927 15:11:14.658645 24238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:14.658059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:43879)
0927 15:11:14.658485 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:14.658542 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":110,"threads_started":1}
I20260927 15:11:15.597972 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:15.597493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.193:55729)
0927 15:11:15.597800 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:15.597871 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":95,"threads_started":1}
I20260927 15:11:15.659986 24241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:15.659501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.195:35937)
0927 15:11:15.659810 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:15.659880 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":101,"threads_started":1}
I20260927 15:11:15.660025 24242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:15.659501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:52423)
0927 15:11:15.659879 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:15.659937 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":103,"threads_started":1}
I20260927 15:11:16.599401 24243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:16.598941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.193:59699)
0927 15:11:16.599230 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:16.599294 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":80,"threads_started":1}
I20260927 15:11:16.661510 24244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:16.661042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.195:36061)
0927 15:11:16.661347 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:16.661412 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":107,"threads_started":1}
I20260927 15:11:16.661545 24245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:16.661098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46765 (local address 127.7.17.194:33569)
0927 15:11:16.661405 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:16.661458 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.254:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":63,"threads_started":1}
I20260927 15:11:16.729388 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 23736
I20260927 15:11:16.734637 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 23910
I20260927 15:11:16.739864 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 24045
2026-09-27T15:11:16Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59437 ms)
[----------] 30 tests from SecurityITest (240720 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:16 dist-test-slave-hhpq krb5kdc[24248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:16 dist-test-slave-hhpq krb5kdc[24248](info): set up 2 sockets
Sep 27 15:11:16 dist-test-slave-hhpq krb5kdc[24248](info): commencing operation
krb5kdc: starting...
W20260927 15:11:17.716454 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:11:17 dist-test-slave-hhpq krb5kdc[24248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:11:17.840036 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:39085
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:39085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:17.950543 24261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:17.950845 24261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:17.950899 24261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:17.954990 24261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:17.955065 24261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:17.955101 24261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:17.955122 24261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:17.955152 24261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:17.960532 24261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:39085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:39085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:17.961683 24261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:17.962721 24261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:17.971544 24267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:17.971544 24266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:17.971520 24261 server_base.cc:1034] running on GCE node
W20260927 15:11:17.971503 24269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:17.972442 24261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:17.973661 24261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:17.974819 24261 hybrid_clock.cc:648] HybridClock initialized: now 1790521877974797 us; error 39 us; skew 500 ppm
Sep 27 15:11:17 dist-test-slave-hhpq krb5kdc[24248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521877, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:17.977741 24261 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:11:17.978739 24261 webserver.cc:460] Webserver started at http://127.7.17.254:43067/ using document root <none> and password file <none>
I20260927 15:11:17.979352 24261 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:17.979417 24261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:17.979633 24261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:17.981863 24261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "4d5707e313a34bdf9b6d6bb77905536d"
format_stamp: "Formatted at 2026-09-27 15:11:17 on dist-test-slave-hhpq"
server_key: "77b9a150fcb5bbc4ae72608d76497954"
server_key_iv: "39753f67c96c80012418c9635ee3bb7e"
server_key_version: "encryptionkey@0"
I20260927 15:11:17.982448 24261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "4d5707e313a34bdf9b6d6bb77905536d"
format_stamp: "Formatted at 2026-09-27 15:11:17 on dist-test-slave-hhpq"
server_key: "77b9a150fcb5bbc4ae72608d76497954"
server_key_iv: "39753f67c96c80012418c9635ee3bb7e"
server_key_version: "encryptionkey@0"
I20260927 15:11:17.986668 24261 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:11:17.989439 24276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:17.990876 24261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:17.991014 24261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "4d5707e313a34bdf9b6d6bb77905536d"
format_stamp: "Formatted at 2026-09-27 15:11:17 on dist-test-slave-hhpq"
server_key: "77b9a150fcb5bbc4ae72608d76497954"
server_key_iv: "39753f67c96c80012418c9635ee3bb7e"
server_key_version: "encryptionkey@0"
I20260927 15:11:17.991156 24261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:18.015661 24261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:18.018793 24261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:18.018977 24261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:18.025871 24261 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:39085
I20260927 15:11:18.025882 24328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:39085 every 8 connection(s)
I20260927 15:11:18.026899 24261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:18.029975 24329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:18.036250 24329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d: Bootstrap starting.
I20260927 15:11:18.036505 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 24261
I20260927 15:11:18.036620 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:18.036973 7239 external_mini_cluster.cc:1428] Setting key 5d938b7ad69f91ee84584aa75c63537e
I20260927 15:11:18.039263 24329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:18.040061 24329 log.cc:824] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:18.042385 24329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d: No bootstrap required, opened a new log
Sep 27 15:11:18 dist-test-slave-hhpq krb5kdc[24248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:11:18.046934 24329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5707e313a34bdf9b6d6bb77905536d" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39085 } }
I20260927 15:11:18.047221 24329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:18.047305 24329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d5707e313a34bdf9b6d6bb77905536d, State: Initialized, Role: FOLLOWER
I20260927 15:11:18.047847 24329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5707e313a34bdf9b6d6bb77905536d" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39085 } }
I20260927 15:11:18.048022 24329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:18.048098 24329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:18.048209 24329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:18.049484 24329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5707e313a34bdf9b6d6bb77905536d" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39085 } }
I20260927 15:11:18.049878 24329 leader_election.cc:302] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d5707e313a34bdf9b6d6bb77905536d; no voters:
I20260927 15:11:18.050268 24329 leader_election.cc:288] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:18.050391 24334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:18.050659 24334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d [term 1 LEADER]: Becoming Leader. State: Replica: 4d5707e313a34bdf9b6d6bb77905536d, State: Running, Role: LEADER
I20260927 15:11:18.051024 24334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5707e313a34bdf9b6d6bb77905536d" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39085 } }
I20260927 15:11:18.051707 24329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:18.052359 24334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d5707e313a34bdf9b6d6bb77905536d. Latest consensus state: current_term: 1 leader_uuid: "4d5707e313a34bdf9b6d6bb77905536d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5707e313a34bdf9b6d6bb77905536d" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39085 } } }
I20260927 15:11:18.052587 24334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:18.052747 24336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d5707e313a34bdf9b6d6bb77905536d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5707e313a34bdf9b6d6bb77905536d" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39085 } } }
I20260927 15:11:18.052922 24336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:18.052976 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:18.038486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42280 (local address 127.7.17.254:39085)
0927 15:11:18.038989 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:18.039006 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:18.039047 (+ 41us) server_negotiation.cc:407] Connection header received
0927 15:11:18.039848 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:18.039871 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:18.040297 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:18.040714 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:18.041585 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.042689 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.043849 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.044384 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.044508 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:18.047380 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:18.047388 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:18.047409 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:18.047447 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:18.049442 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:18.049948 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:18.049955 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:18.049962 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:18.050116 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:18.050462 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:18.050466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:18.050467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:18.051010 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:18.051214 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:18.051518 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 15:11:18.051849 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":157,"threads_started":1}
I20260927 15:11:18.053052 24342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:18.056197 24342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:18.061807 24342 catalog_manager.cc:1347] Generated new cluster ID: dac68eb7d7f1414d95bd2b2a153f53fb
I20260927 15:11:18.061896 24342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:18.078900 24342 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:18.080365 24342 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:18.091118 24342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d: Generated new TSK 0
I20260927 15:11:18.092100 24342 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
I20260927 15:11:18.128422 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39085
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:18.240782 24355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:18.241106 24355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:18.241153 24355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:18.245329 24355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:18.245399 24355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:18.245472 24355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:11:18.250982 24355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:18.252097 24355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:18.253216 24355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:18.263053 24361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:18.263041 24360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:18.263041 24363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:18.263378 24355 server_base.cc:1034] running on GCE node
I20260927 15:11:18.263823 24355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:18.264509 24355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:18.265671 24355 hybrid_clock.cc:648] HybridClock initialized: now 1790521878265642 us; error 46 us; skew 500 ppm
Sep 27 15:11:18 dist-test-slave-hhpq krb5kdc[24248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521878, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:18.268573 24355 init.cc:377] Logged in from keytab as kudu/127.7.17.193@KRBTEST.COM (short username kudu)
I20260927 15:11:18.269531 24355 webserver.cc:460] Webserver started at http://127.7.17.193:40681/ using document root <none> and password file <none>
I20260927 15:11:18.270097 24355 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:18.270157 24355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:18.270352 24355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:18.272504 24355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "720600894aa745429688f57962cbde7a"
format_stamp: "Formatted at 2026-09-27 15:11:18 on dist-test-slave-hhpq"
server_key: "fbdf27a1669e778023df066549a98740"
server_key_iv: "4f7b2ff791bc103794e0c8c513025c3f"
server_key_version: "encryptionkey@0"
I20260927 15:11:18.273123 24355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "720600894aa745429688f57962cbde7a"
format_stamp: "Formatted at 2026-09-27 15:11:18 on dist-test-slave-hhpq"
server_key: "fbdf27a1669e778023df066549a98740"
server_key_iv: "4f7b2ff791bc103794e0c8c513025c3f"
server_key_version: "encryptionkey@0"
I20260927 15:11:18.277014 24355 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:11:18.279663 24370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:18.281107 24355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:18.281219 24355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "720600894aa745429688f57962cbde7a"
format_stamp: "Formatted at 2026-09-27 15:11:18 on dist-test-slave-hhpq"
server_key: "fbdf27a1669e778023df066549a98740"
server_key_iv: "4f7b2ff791bc103794e0c8c513025c3f"
server_key_version: "encryptionkey@0"
I20260927 15:11:18.281340 24355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:18.303390 24355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:18.306759 24355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:18.306953 24355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:18.307641 24355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:18.308533 24355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:18.308584 24355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:18.308625 24355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:18.308641 24355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:18.317406 24355 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:34957
I20260927 15:11:18.317515 24483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:34957 every 8 connection(s)
I20260927 15:11:18.318397 24355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:11:18.325121 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 24355
I20260927 15:11:18.325294 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:11:18.325570 7239 external_mini_cluster.cc:1428] Setting key d1f50d8b4cb45daa09f52c4f6383ad6a
Sep 27 15:11:18 dist-test-slave-hhpq krb5kdc[24248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521878, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:11:18.333309 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:18.320361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:41257 (local address 127.7.17.254:39085)
0927 15:11:18.320536 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:18.320541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:18.321588 (+ 1047us) server_negotiation.cc:407] Connection header received
0927 15:11:18.322885 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:18.322888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:18.322941 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:18.323024 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:18.324333 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.324897 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.326378 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.326691 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.326784 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:18.329668 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:18.329672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:18.329674 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:18.329704 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:18.331103 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:18.331697 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:18.331700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:18.331701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:18.331784 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:18.332123 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:18.332126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:18.332128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:18.332324 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:18.332411 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:18.332979 (+ 568us) server_negotiation.cc:299] Negotiation successful
0927 15:11:18.333132 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":69}
I20260927 15:11:18.334185 24486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:18.320871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:39085 (local address 127.7.17.193:41257)
0927 15:11:18.321404 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:18.321456 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:18.322637 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:18.323196 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:18.323241 (+ 45us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:18.323624 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:18.324160 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:18.324176 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.325068 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.325071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:18.326248 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:18.326260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.326865 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.326868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:18.327086 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:18.327816 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:18.327833 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:18.329452 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:18.331278 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:18.331284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:18.331300 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:18.331591 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:18.331871 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:18.331874 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:18.331876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:18.332034 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:18.332429 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:18.332436 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:18.332718 (+ 282us) client_negotiation.cc:769] Sending connection context
0927 15:11:18.332928 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 15:11:18.333322 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.7.17.194@KRBTEST.COM; defaulting to no policy
I20260927 15:11:18.335448 24484 heartbeater.cc:347] Connected to a master server at 127.7.17.254:39085
I20260927 15:11:18.335708 24484 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:18.336270 24484 heartbeater.cc:511] Master 127.7.17.254:39085 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.194@KRBTEST.COM" created.
I20260927 15:11:18.337993 24292 ts_manager.cc:194] Registered new tserver with Master: 720600894aa745429688f57962cbde7a (127.7.17.193:34957)
I20260927 15:11:18.339184 24292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:41257
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
I20260927 15:11:18.354066 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39085
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:18.470119 24489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:18.470407 24489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:18.470451 24489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:18.474776 24489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:18.474860 24489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:18.474925 24489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:11:18.480753 24489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:18.481950 24489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:18.483026 24489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:18.493219 24494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:18.493258 24495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:18.493382 24489 server_base.cc:1034] running on GCE node
W20260927 15:11:18.493598 24497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:18.494057 24489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:18.494750 24489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:18.495936 24489 hybrid_clock.cc:648] HybridClock initialized: now 1790521878495900 us; error 49 us; skew 500 ppm
Sep 27 15:11:18 dist-test-slave-hhpq krb5kdc[24248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521878, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:18.498920 24489 init.cc:377] Logged in from keytab as kudu/127.7.17.194@KRBTEST.COM (short username kudu)
I20260927 15:11:18.499954 24489 webserver.cc:460] Webserver started at http://127.7.17.194:44567/ using document root <none> and password file <none>
I20260927 15:11:18.500535 24489 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:18.500599 24489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:18.500805 24489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:18.503011 24489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "d18e62d719d74ec1a80c2b529f5de579"
format_stamp: "Formatted at 2026-09-27 15:11:18 on dist-test-slave-hhpq"
server_key: "6c087f4a999710f0f27746776f20e73c"
server_key_iv: "3f105b413bcb252faef0d5b58c3335c0"
server_key_version: "encryptionkey@0"
I20260927 15:11:18.503578 24489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "d18e62d719d74ec1a80c2b529f5de579"
format_stamp: "Formatted at 2026-09-27 15:11:18 on dist-test-slave-hhpq"
server_key: "6c087f4a999710f0f27746776f20e73c"
server_key_iv: "3f105b413bcb252faef0d5b58c3335c0"
server_key_version: "encryptionkey@0"
I20260927 15:11:18.507441 24489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:11:18.510079 24504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:18.511369 24489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:11:18.511474 24489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "d18e62d719d74ec1a80c2b529f5de579"
format_stamp: "Formatted at 2026-09-27 15:11:18 on dist-test-slave-hhpq"
server_key: "6c087f4a999710f0f27746776f20e73c"
server_key_iv: "3f105b413bcb252faef0d5b58c3335c0"
server_key_version: "encryptionkey@0"
I20260927 15:11:18.511569 24489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:18.530040 24489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:18.533298 24489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:18.533480 24489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:18.534122 24489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:18.535054 24489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:18.535104 24489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:18.535144 24489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:18.535171 24489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:18.544260 24489 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:42547
I20260927 15:11:18.544282 24617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:42547 every 8 connection(s)
I20260927 15:11:18.545252 24489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:11:18.550737 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 24489
I20260927 15:11:18.550885 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:11:18.551203 7239 external_mini_cluster.cc:1428] Setting key 46225560b3bd3adad85d6c5d450acd16
Sep 27 15:11:18 dist-test-slave-hhpq krb5kdc[24248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521878, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.17.195@KRBTEST.COM; defaulting to no policy
I20260927 15:11:18.561599 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:18.547308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:48023 (local address 127.7.17.254:39085)
0927 15:11:18.547437 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:18.547440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:18.548327 (+ 887us) server_negotiation.cc:407] Connection header received
0927 15:11:18.549832 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:18.549836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:18.549899 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:18.549974 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:18.551700 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.552422 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.554070 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.554395 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.554494 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:18.557648 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:18.557652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:18.557654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:18.557684 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:18.559080 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:18.559822 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:18.559825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:18.559826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:18.559905 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:18.560303 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:18.560305 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:18.560306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:18.560490 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:18.560563 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:18.561348 (+ 785us) server_negotiation.cc:299] Negotiation successful
0927 15:11:18.561467 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:11:18.562497 24620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:18.547631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:39085 (local address 127.7.17.194:48023)
0927 15:11:18.548144 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:18.548195 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:18.549574 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:18.550188 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:18.550196 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:18.550701 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:18.551467 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:18.551490 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.552583 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.552587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:18.553921 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:18.553936 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.554574 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.554578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:18.554820 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:18.555502 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:18.555523 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:18.557415 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:18.559253 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:18.559267 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:18.559289 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:18.559712 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:18.559993 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:18.559996 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:18.559998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:18.560212 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:18.560609 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:18.560617 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:18.561038 (+ 421us) client_negotiation.cc:769] Sending connection context
0927 15:11:18.561254 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 15:11:18.561558 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.195@KRBTEST.COM" created.
I20260927 15:11:18.563959 24618 heartbeater.cc:347] Connected to a master server at 127.7.17.254:39085
I20260927 15:11:18.564347 24618 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:18.565084 24618 heartbeater.cc:511] Master 127.7.17.254:39085 requested a full tablet report, sending...
I20260927 15:11:18.566473 24292 ts_manager.cc:194] Registered new tserver with Master: d18e62d719d74ec1a80c2b529f5de579 (127.7.17.194:42547)
I20260927 15:11:18.567250 24292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:48023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
I20260927 15:11:18.580718 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39085
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:33797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:18.693207 24623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:18.693487 24623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:18.693529 24623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:18.697806 24623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:18.697888 24623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:18.697952 24623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:11:18.703533 24623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:33797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:18.704665 24623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:18.705726 24623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:18.715204 24628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:18.715267 24631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:18.715193 24629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:18.715407 24623 server_base.cc:1034] running on GCE node
I20260927 15:11:18.715971 24623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:18.716656 24623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:18.717813 24623 hybrid_clock.cc:648] HybridClock initialized: now 1790521878717791 us; error 35 us; skew 500 ppm
Sep 27 15:11:18 dist-test-slave-hhpq krb5kdc[24248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521878, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:18.720449 24623 init.cc:377] Logged in from keytab as kudu/127.7.17.195@KRBTEST.COM (short username kudu)
I20260927 15:11:18.721392 24623 webserver.cc:460] Webserver started at http://127.7.17.195:42219/ using document root <none> and password file <none>
I20260927 15:11:18.721913 24623 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:18.721971 24623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:18.722154 24623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:18.724156 24623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "94cccc485f15424c91b7d95ffc9d4682"
format_stamp: "Formatted at 2026-09-27 15:11:18 on dist-test-slave-hhpq"
server_key: "479c20102095f60246b46962c49351a7"
server_key_iv: "9d28451b3a51fd6da61b8104885290ef"
server_key_version: "encryptionkey@0"
I20260927 15:11:18.724671 24623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "94cccc485f15424c91b7d95ffc9d4682"
format_stamp: "Formatted at 2026-09-27 15:11:18 on dist-test-slave-hhpq"
server_key: "479c20102095f60246b46962c49351a7"
server_key_iv: "9d28451b3a51fd6da61b8104885290ef"
server_key_version: "encryptionkey@0"
I20260927 15:11:18.728487 24623 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:11:18.730948 24638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:18.732256 24623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:11:18.732355 24623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "94cccc485f15424c91b7d95ffc9d4682"
format_stamp: "Formatted at 2026-09-27 15:11:18 on dist-test-slave-hhpq"
server_key: "479c20102095f60246b46962c49351a7"
server_key_iv: "9d28451b3a51fd6da61b8104885290ef"
server_key_version: "encryptionkey@0"
I20260927 15:11:18.732445 24623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:18.748960 24623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:18.752136 24623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:18.752323 24623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:18.752955 24623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:18.753841 24623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:18.753894 24623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:18.753937 24623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:18.753965 24623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:18.762785 24623 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:39779
I20260927 15:11:18.762822 24751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:39779 every 8 connection(s)
I20260927 15:11:18.763722 24623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:11:18.767094 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 24623
I20260927 15:11:18.767212 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:11:18.767515 7239 external_mini_cluster.cc:1428] Setting key 6db60a3a0abfdc286c9e4348eeb97b8d
Sep 27 15:11:18 dist-test-slave-hhpq krb5kdc[24248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521878, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:11:18.778363 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:18.765662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:43035 (local address 127.7.17.254:39085)
0927 15:11:18.765814 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:18.765818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:18.766692 (+ 874us) server_negotiation.cc:407] Connection header received
0927 15:11:18.768134 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:18.768137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:18.768199 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:18.768279 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:18.769913 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.770523 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.771689 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.771919 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.772015 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:18.774671 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:18.774674 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:18.774676 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:18.774703 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:18.776113 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:18.776603 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:18.776606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:18.776607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:18.776694 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:18.777241 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:18.777244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:18.777245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:18.777442 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:18.777529 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:18.778121 (+ 592us) server_negotiation.cc:299] Negotiation successful
0927 15:11:18.778235 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:11:18.779287 24754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:18.766051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:39085 (local address 127.7.17.195:43035)
0927 15:11:18.766524 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:18.766573 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:18.767865 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:18.768674 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:18.768684 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:18.769161 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:18.769765 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:18.769781 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.770627 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.770629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:18.771552 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:18.771563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.772092 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.772095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:18.772274 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:18.772901 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:18.772918 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:18.774476 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:18.776216 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:18.776221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:18.776235 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:18.776497 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:18.776865 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:18.776868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:18.776870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:18.777033 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:18.777561 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:18.777567 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:18.777843 (+ 276us) client_negotiation.cc:769] Sending connection context
0927 15:11:18.778010 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 15:11:18.778335 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":112,"threads_started":1}
I20260927 15:11:18.780646 24752 heartbeater.cc:347] Connected to a master server at 127.7.17.254:39085
I20260927 15:11:18.780922 24752 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:18.781500 24752 heartbeater.cc:511] Master 127.7.17.254:39085 requested a full tablet report, sending...
I20260927 15:11:18.782526 24292 ts_manager.cc:194] Registered new tserver with Master: 94cccc485f15424c91b7d95ffc9d4682 (127.7.17.195:39779)
I20260927 15:11:18.783119 24292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:43035
I20260927 15:11:18.793556 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:11:18.802551 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:18.795464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42284 (local address 127.7.17.254:39085)
0927 15:11:18.795580 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:18.795585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:18.795800 (+ 215us) server_negotiation.cc:407] Connection header received
0927 15:11:18.795919 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:18.795922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:18.795969 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:18.796062 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:18.796751 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.797308 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.798123 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.798344 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.798421 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:18.799746 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:18.799749 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:18.799751 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:18.799779 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:18.801083 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:18.801486 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:18.801489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:18.801490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:18.801562 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:18.801817 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:18.801819 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:18.801820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:18.802038 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:18.802140 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:18.802328 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 15:11:18.802425 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:11:18.805603 24292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:11:18.808650 24292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:11:18.823748 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:18.818304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:42547 (local address 127.0.0.1:38902)
0927 15:11:18.818950 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:18.818963 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:18.819060 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:18.819700 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:18.819703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:18.819719 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:18.819801 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:18.819807 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.821536 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.821539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:18.822328 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:18.822339 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.823135 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.823139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:18.823297 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:18.823336 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 15:11:18.823445 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 15:11:18.823586 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":46,"thread_start_us":82,"threads_started":1}
I20260927 15:11:18.823760 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:18.818453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38902 (local address 127.7.17.194:42547)
0927 15:11:18.818868 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:18.818878 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:18.819057 (+ 179us) server_negotiation.cc:407] Connection header received
0927 15:11:18.819224 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:18.819228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:18.819383 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:18.819513 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:18.819920 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.821432 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.822446 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.823014 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.823114 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:18.823273 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:18.823496 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 15:11:18.823604 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":104,"threads_started":1}
I20260927 15:11:18.823813 24764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:18.817719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46360 (local address 127.7.17.193:34957)
0927 15:11:18.818416 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:18.818432 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:18.818830 (+ 398us) server_negotiation.cc:407] Connection header received
0927 15:11:18.819036 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:18.819043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:18.819265 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:18.819399 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:18.819692 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.820883 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.822372 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.822999 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.823095 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:18.823256 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:18.823581 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 15:11:18.823688 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":67,"threads_started":1}
I20260927 15:11:18.824338 24765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:18.817537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:34957 (local address 127.0.0.1:46360)
0927 15:11:18.818667 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:18.818725 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:18.818864 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:18.819397 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:18.819403 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:18.819424 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:18.819566 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:18.819573 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.821025 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.821029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:18.822241 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:18.822253 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.823120 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.823123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:18.823291 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:18.823334 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 15:11:18.823448 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 15:11:18.823991 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"mutex_wait_us":468,"thread_start_us":87,"threads_started":1}
I20260927 15:11:18.825712 24763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:18.817695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:39779 (local address 127.0.0.1:50128)
0927 15:11:18.818789 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:18.818800 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:18.818864 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:18.819585 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:18.819588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:18.819604 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:18.819690 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:18.819695 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.820678 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.820682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:18.822261 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:18.822267 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.825320 (+ 3053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.825323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:18.825412 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:18.825434 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:11:18.825508 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 15:11:18.825582 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"mutex_wait_us":63,"thread_start_us":124,"threads_started":1}
I20260927 15:11:18.825783 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:18.817997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50128 (local address 127.7.17.195:39779)
0927 15:11:18.818669 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:18.818681 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:18.819000 (+ 319us) server_negotiation.cc:407] Connection header received
0927 15:11:18.819081 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:18.819086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:18.819270 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:18.819535 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:18.819795 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.820560 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.823318 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:18.824548 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:18.825313 (+ 765us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:18.825459 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:18.825548 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 15:11:18.825653 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":81,"threads_started":1}
I20260927 15:11:18.827224 24552 tablet_service.cc:1467] Processing CreateTablet for tablet 17db5066d9464d8c93b9ae56567ee14a (DEFAULT_TABLE table=test-table [id=3d8da8f278244d1fa09c34d804f26194]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:18.827225 24418 tablet_service.cc:1467] Processing CreateTablet for tablet 17db5066d9464d8c93b9ae56567ee14a (DEFAULT_TABLE table=test-table [id=3d8da8f278244d1fa09c34d804f26194]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:18.827953 24686 tablet_service.cc:1467] Processing CreateTablet for tablet 17db5066d9464d8c93b9ae56567ee14a (DEFAULT_TABLE table=test-table [id=3d8da8f278244d1fa09c34d804f26194]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:18.828380 24418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17db5066d9464d8c93b9ae56567ee14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:18.828380 24552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17db5066d9464d8c93b9ae56567ee14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:18.829041 24686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17db5066d9464d8c93b9ae56567ee14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:18.835677 24769 tablet_bootstrap.cc:492] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682: Bootstrap starting.
I20260927 15:11:18.836369 24770 tablet_bootstrap.cc:492] T 17db5066d9464d8c93b9ae56567ee14a P 720600894aa745429688f57962cbde7a: Bootstrap starting.
I20260927 15:11:18.838379 24770 tablet_bootstrap.cc:654] T 17db5066d9464d8c93b9ae56567ee14a P 720600894aa745429688f57962cbde7a: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:18.838380 24769 tablet_bootstrap.cc:654] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:18.839193 24770 log.cc:824] T 17db5066d9464d8c93b9ae56567ee14a P 720600894aa745429688f57962cbde7a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:18.839759 24769 log.cc:824] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:18.839903 24771 tablet_bootstrap.cc:492] T 17db5066d9464d8c93b9ae56567ee14a P d18e62d719d74ec1a80c2b529f5de579: Bootstrap starting.
I20260927 15:11:18.841351 24770 tablet_bootstrap.cc:492] T 17db5066d9464d8c93b9ae56567ee14a P 720600894aa745429688f57962cbde7a: No bootstrap required, opened a new log
I20260927 15:11:18.841564 24770 ts_tablet_manager.cc:1397] T 17db5066d9464d8c93b9ae56567ee14a P 720600894aa745429688f57962cbde7a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:11:18.841785 24769 tablet_bootstrap.cc:492] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682: No bootstrap required, opened a new log
I20260927 15:11:18.841967 24769 ts_tablet_manager.cc:1397] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:11:18.842299 24771 tablet_bootstrap.cc:654] T 17db5066d9464d8c93b9ae56567ee14a P d18e62d719d74ec1a80c2b529f5de579: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:18.842945 24771 log.cc:824] T 17db5066d9464d8c93b9ae56567ee14a P d18e62d719d74ec1a80c2b529f5de579: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:18.844602 24771 tablet_bootstrap.cc:492] T 17db5066d9464d8c93b9ae56567ee14a P d18e62d719d74ec1a80c2b529f5de579: No bootstrap required, opened a new log
I20260927 15:11:18.844791 24771 ts_tablet_manager.cc:1397] T 17db5066d9464d8c93b9ae56567ee14a P d18e62d719d74ec1a80c2b529f5de579: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:11:18.845821 24770 raft_consensus.cc:348] T 17db5066d9464d8c93b9ae56567ee14a P 720600894aa745429688f57962cbde7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720600894aa745429688f57962cbde7a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 34957 } } peers { permanent_uuid: "94cccc485f15424c91b7d95ffc9d4682" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 39779 } } peers { permanent_uuid: "d18e62d719d74ec1a80c2b529f5de579" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42547 } }
I20260927 15:11:18.845821 24769 raft_consensus.cc:348] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720600894aa745429688f57962cbde7a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 34957 } } peers { permanent_uuid: "94cccc485f15424c91b7d95ffc9d4682" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 39779 } } peers { permanent_uuid: "d18e62d719d74ec1a80c2b529f5de579" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42547 } }
I20260927 15:11:18.846208 24769 raft_consensus.cc:374] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:18.846208 24770 raft_consensus.cc:374] T 17db5066d9464d8c93b9ae56567ee14a P 720600894aa745429688f57962cbde7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:18.846268 24769 raft_consensus.cc:729] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94cccc485f15424c91b7d95ffc9d4682, State: Initialized, Role: FOLLOWER
I20260927 15:11:18.846268 24770 raft_consensus.cc:729] T 17db5066d9464d8c93b9ae56567ee14a P 720600894aa745429688f57962cbde7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720600894aa745429688f57962cbde7a, State: Initialized, Role: FOLLOWER
I20260927 15:11:18.846745 24770 consensus_queue.cc:260] T 17db5066d9464d8c93b9ae56567ee14a P 720600894aa745429688f57962cbde7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720600894aa745429688f57962cbde7a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 34957 } } peers { permanent_uuid: "94cccc485f15424c91b7d95ffc9d4682" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 39779 } } peers { permanent_uuid: "d18e62d719d74ec1a80c2b529f5de579" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42547 } }
I20260927 15:11:18.847113 24769 consensus_queue.cc:260] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720600894aa745429688f57962cbde7a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 34957 } } peers { permanent_uuid: "94cccc485f15424c91b7d95ffc9d4682" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 39779 } } peers { permanent_uuid: "d18e62d719d74ec1a80c2b529f5de579" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42547 } }
I20260927 15:11:18.847793 24752 heartbeater.cc:503] Master 127.7.17.254:39085 was elected leader, sending a full tablet report...
I20260927 15:11:18.847834 24770 ts_tablet_manager.cc:1428] T 17db5066d9464d8c93b9ae56567ee14a P 720600894aa745429688f57962cbde7a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:11:18.847983 24769 ts_tablet_manager.cc:1428] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:11:18.848169 24484 heartbeater.cc:503] Master 127.7.17.254:39085 was elected leader, sending a full tablet report...
I20260927 15:11:18.849442 24771 raft_consensus.cc:348] T 17db5066d9464d8c93b9ae56567ee14a P d18e62d719d74ec1a80c2b529f5de579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720600894aa745429688f57962cbde7a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 34957 } } peers { permanent_uuid: "94cccc485f15424c91b7d95ffc9d4682" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 39779 } } peers { permanent_uuid: "d18e62d719d74ec1a80c2b529f5de579" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42547 } }
I20260927 15:11:18.849678 24771 raft_consensus.cc:374] T 17db5066d9464d8c93b9ae56567ee14a P d18e62d719d74ec1a80c2b529f5de579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:18.849733 24771 raft_consensus.cc:729] T 17db5066d9464d8c93b9ae56567ee14a P d18e62d719d74ec1a80c2b529f5de579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d18e62d719d74ec1a80c2b529f5de579, State: Initialized, Role: FOLLOWER
I20260927 15:11:18.850139 24771 consensus_queue.cc:260] T 17db5066d9464d8c93b9ae56567ee14a P d18e62d719d74ec1a80c2b529f5de579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720600894aa745429688f57962cbde7a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 34957 } } peers { permanent_uuid: "94cccc485f15424c91b7d95ffc9d4682" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 39779 } } peers { permanent_uuid: "d18e62d719d74ec1a80c2b529f5de579" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42547 } }
I20260927 15:11:18.850776 24618 heartbeater.cc:503] Master 127.7.17.254:39085 was elected leader, sending a full tablet report...
I20260927 15:11:18.851095 24771 ts_tablet_manager.cc:1428] T 17db5066d9464d8c93b9ae56567ee14a P d18e62d719d74ec1a80c2b529f5de579: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
W20260927 15:11:19.014694 24753 tablet.cc:2369] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:11:19.046419 24619 tablet.cc:2369] T 17db5066d9464d8c93b9ae56567ee14a P d18e62d719d74ec1a80c2b529f5de579: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:11:19.069809 24485 tablet.cc:2369] T 17db5066d9464d8c93b9ae56567ee14a P 720600894aa745429688f57962cbde7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:11:19.106988 24776 raft_consensus.cc:482] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:11:19.107215 24776 raft_consensus.cc:504] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720600894aa745429688f57962cbde7a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 34957 } } peers { permanent_uuid: "94cccc485f15424c91b7d95ffc9d4682" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 39779 } } peers { permanent_uuid: "d18e62d719d74ec1a80c2b529f5de579" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42547 } }
I20260927 15:11:19.108542 24776 leader_election.cc:288] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 720600894aa745429688f57962cbde7a (127.7.17.193:34957), d18e62d719d74ec1a80c2b529f5de579 (127.7.17.194:42547)
I20260927 15:11:19.112336 24754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:19.108791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:42547 (local address 127.7.17.195:37071)
0927 15:11:19.108937 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:19.108951 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:19.109038 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:19.109317 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:19.109320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:19.109339 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:19.109451 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:19.109458 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:19.110280 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:19.110283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:19.111137 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:19.111147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:19.111877 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:19.111881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:19.111984 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:19.112010 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:11:19.112100 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 15:11:19.112174 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 15:11:19.112339 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:19.108996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:37071 (local address 127.7.17.194:42547)
0927 15:11:19.109115 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:19.109119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:19.109132 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:11:19.109178 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:19.109180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:19.109221 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:19.109294 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:19.109578 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:19.110180 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:19.111275 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:19.111778 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:19.111868 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:19.111908 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:19.112128 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 15:11:19.112211 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 15:11:19.112628 24778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:19.108792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:34957 (local address 127.7.17.195:44023)
0927 15:11:19.109359 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:19.109370 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:19.109450 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:19.109656 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:19.109658 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:19.109673 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:19.109788 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:19.109794 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:19.110538 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:19.110541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:19.111452 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:19.111462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:19.112212 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:19.112215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:19.112311 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:19.112332 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:11:19.112406 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 15:11:19.112466 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":257,"thread_start_us":116,"threads_started":1}
I20260927 15:11:19.112926 24764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:19.108960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:44023 (local address 127.7.17.193:34957)
0927 15:11:19.109081 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:19.109085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:19.109443 (+ 358us) server_negotiation.cc:407] Connection header received
0927 15:11:19.109523 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:19.109526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:19.109578 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:19.109631 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:19.109878 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:19.110416 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:19.111574 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:19.112076 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:19.112172 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:19.112215 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:19.112699 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 15:11:19.112791 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:11:19.113034 24572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17db5066d9464d8c93b9ae56567ee14a" candidate_uuid: "94cccc485f15424c91b7d95ffc9d4682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d18e62d719d74ec1a80c2b529f5de579" is_pre_election: true
I20260927 15:11:19.113400 24572 raft_consensus.cc:2454] T 17db5066d9464d8c93b9ae56567ee14a P d18e62d719d74ec1a80c2b529f5de579 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94cccc485f15424c91b7d95ffc9d4682 in term 0.
I20260927 15:11:19.113493 24438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17db5066d9464d8c93b9ae56567ee14a" candidate_uuid: "94cccc485f15424c91b7d95ffc9d4682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720600894aa745429688f57962cbde7a" is_pre_election: true
I20260927 15:11:19.113770 24438 raft_consensus.cc:2454] T 17db5066d9464d8c93b9ae56567ee14a P 720600894aa745429688f57962cbde7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94cccc485f15424c91b7d95ffc9d4682 in term 0.
I20260927 15:11:19.113905 24641 leader_election.cc:302] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94cccc485f15424c91b7d95ffc9d4682, d18e62d719d74ec1a80c2b529f5de579; no voters:
I20260927 15:11:19.114174 24776 raft_consensus.cc:2781] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:11:19.114249 24776 raft_consensus.cc:482] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:11:19.114279 24776 raft_consensus.cc:3037] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:19.115325 24776 raft_consensus.cc:504] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720600894aa745429688f57962cbde7a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 34957 } } peers { permanent_uuid: "94cccc485f15424c91b7d95ffc9d4682" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 39779 } } peers { permanent_uuid: "d18e62d719d74ec1a80c2b529f5de579" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42547 } }
I20260927 15:11:19.115649 24776 leader_election.cc:288] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682 [CANDIDATE]: Term 1 election: Requested vote from peers 720600894aa745429688f57962cbde7a (127.7.17.193:34957), d18e62d719d74ec1a80c2b529f5de579 (127.7.17.194:42547)
I20260927 15:11:19.115973 24438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17db5066d9464d8c93b9ae56567ee14a" candidate_uuid: "94cccc485f15424c91b7d95ffc9d4682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720600894aa745429688f57962cbde7a"
I20260927 15:11:19.116083 24438 raft_consensus.cc:3037] T 17db5066d9464d8c93b9ae56567ee14a P 720600894aa745429688f57962cbde7a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:19.116976 24572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17db5066d9464d8c93b9ae56567ee14a" candidate_uuid: "94cccc485f15424c91b7d95ffc9d4682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d18e62d719d74ec1a80c2b529f5de579"
I20260927 15:11:19.117098 24572 raft_consensus.cc:3037] T 17db5066d9464d8c93b9ae56567ee14a P d18e62d719d74ec1a80c2b529f5de579 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:19.117131 24438 raft_consensus.cc:2454] T 17db5066d9464d8c93b9ae56567ee14a P 720600894aa745429688f57962cbde7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94cccc485f15424c91b7d95ffc9d4682 in term 1.
I20260927 15:11:19.117457 24639 leader_election.cc:302] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 720600894aa745429688f57962cbde7a, 94cccc485f15424c91b7d95ffc9d4682; no voters:
I20260927 15:11:19.117609 24776 raft_consensus.cc:2781] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:19.117808 24776 raft_consensus.cc:686] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682 [term 1 LEADER]: Becoming Leader. State: Replica: 94cccc485f15424c91b7d95ffc9d4682, State: Running, Role: LEADER
I20260927 15:11:19.118127 24776 consensus_queue.cc:237] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720600894aa745429688f57962cbde7a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 34957 } } peers { permanent_uuid: "94cccc485f15424c91b7d95ffc9d4682" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 39779 } } peers { permanent_uuid: "d18e62d719d74ec1a80c2b529f5de579" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42547 } }
I20260927 15:11:19.118198 24572 raft_consensus.cc:2454] T 17db5066d9464d8c93b9ae56567ee14a P d18e62d719d74ec1a80c2b529f5de579 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94cccc485f15424c91b7d95ffc9d4682 in term 1.
I20260927 15:11:19.121049 24287 catalog_manager.cc:5495] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94cccc485f15424c91b7d95ffc9d4682 (127.7.17.195). New cstate: current_term: 1 leader_uuid: "94cccc485f15424c91b7d95ffc9d4682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720600894aa745429688f57962cbde7a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 34957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94cccc485f15424c91b7d95ffc9d4682" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 39779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d18e62d719d74ec1a80c2b529f5de579" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42547 } health_report { overall_health: UNKNOWN } } }
I20260927 15:11:19.187963 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:19.184293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50142 (local address 127.7.17.195:39779)
0927 15:11:19.184429 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:19.184433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:19.184461 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:11:19.184565 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:19.184567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:19.184628 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:19.184721 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:11:19.184993 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:19.185528 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:19.186457 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:19.186674 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:19.186763 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:19.187086 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:11:19.187548 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:11:19.187663 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:19.187727 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 15:11:19.187790 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:11:19.194192 24438 raft_consensus.cc:1261] T 17db5066d9464d8c93b9ae56567ee14a P 720600894aa745429688f57962cbde7a [term 1 FOLLOWER]: Refusing update from remote peer 94cccc485f15424c91b7d95ffc9d4682: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:19.194192 24572 raft_consensus.cc:1261] T 17db5066d9464d8c93b9ae56567ee14a P d18e62d719d74ec1a80c2b529f5de579 [term 1 FOLLOWER]: Refusing update from remote peer 94cccc485f15424c91b7d95ffc9d4682: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:19.195031 24779 consensus_queue.cc:1035] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d18e62d719d74ec1a80c2b529f5de579" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:19.195284 24776 consensus_queue.cc:1035] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682 [LEADER]: Connected to new peer: Peer: permanent_uuid: "720600894aa745429688f57962cbde7a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 34957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:19.206605 24784 mvcc.cc:205] Tried to move back new op lower bound from 7333977617171505152 to 7333977616870555648. Current Snapshot: MvccSnapshot[applied={T|T < 7333977617171505152}]
I20260927 15:11:19.210846 24287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:19.211014 24287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:19.213407 24287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d5707e313a34bdf9b6d6bb77905536d: Sending DeleteTablet for 3 replicas of tablet 17db5066d9464d8c93b9ae56567ee14a
I20260927 15:11:19.214320 24418 tablet_service.cc:1514] Processing DeleteTablet for tablet 17db5066d9464d8c93b9ae56567ee14a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:19 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:46360
I20260927 15:11:19.214339 24552 tablet_service.cc:1514] Processing DeleteTablet for tablet 17db5066d9464d8c93b9ae56567ee14a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:19 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:38902
I20260927 15:11:19.214448 24686 tablet_service.cc:1514] Processing DeleteTablet for tablet 17db5066d9464d8c93b9ae56567ee14a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:19 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:50128
I20260927 15:11:19.215051 24793 tablet_replica.cc:330] T 17db5066d9464d8c93b9ae56567ee14a P d18e62d719d74ec1a80c2b529f5de579: stopping tablet replica
I20260927 15:11:19.215445 24793 raft_consensus.cc:2229] T 17db5066d9464d8c93b9ae56567ee14a P d18e62d719d74ec1a80c2b529f5de579 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:11:19.215492 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 24355
I20260927 15:11:19.215819 24793 raft_consensus.cc:2258] T 17db5066d9464d8c93b9ae56567ee14a P d18e62d719d74ec1a80c2b529f5de579 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:19.216266 24792 tablet_replica.cc:330] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682: stopping tablet replica
I20260927 15:11:19.216547 24792 raft_consensus.cc:2229] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:11:19.216917 24792 raft_consensus.cc:2258] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:19.217665 24793 ts_tablet_manager.cc:1905] T 17db5066d9464d8c93b9ae56567ee14a P d18e62d719d74ec1a80c2b529f5de579: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:19.218263 24792 ts_tablet_manager.cc:1905] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:19.220417 24793 ts_tablet_manager.cc:1918] T 17db5066d9464d8c93b9ae56567ee14a P d18e62d719d74ec1a80c2b529f5de579: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:19.220508 24793 log.cc:1196] T 17db5066d9464d8c93b9ae56567ee14a P d18e62d719d74ec1a80c2b529f5de579: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-1/wal/wals/17db5066d9464d8c93b9ae56567ee14a
I20260927 15:11:19.220796 24793 ts_tablet_manager.cc:1939] T 17db5066d9464d8c93b9ae56567ee14a P d18e62d719d74ec1a80c2b529f5de579: Deleting consensus metadata
I20260927 15:11:19.220861 24792 ts_tablet_manager.cc:1918] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:19.220943 24792 log.cc:1196] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/17db5066d9464d8c93b9ae56567ee14a
I20260927 15:11:19.221263 24792 ts_tablet_manager.cc:1939] T 17db5066d9464d8c93b9ae56567ee14a P 94cccc485f15424c91b7d95ffc9d4682: Deleting consensus metadata
I20260927 15:11:19.222153 24279 catalog_manager.cc:4841] TS d18e62d719d74ec1a80c2b529f5de579 (127.7.17.194:42547): tablet 17db5066d9464d8c93b9ae56567ee14a (table test-table [id=3d8da8f278244d1fa09c34d804f26194]) successfully deleted
I20260927 15:11:19.222349 24279 catalog_manager.cc:4841] TS 94cccc485f15424c91b7d95ffc9d4682 (127.7.17.195:39779): tablet 17db5066d9464d8c93b9ae56567ee14a (table test-table [id=3d8da8f278244d1fa09c34d804f26194]) successfully deleted
W20260927 15:11:19.224329 24277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.17.193:34957 (error 108)
I20260927 15:11:19.225163 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 24489
I20260927 15:11:19.225451 24763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:19.225133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:34957 (local address 127.0.0.1:46364)
0927 15:11:19.225289 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:19.225362 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.193:34957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 15:11:19.226922 24277 catalog_manager.cc:4584] TS 720600894aa745429688f57962cbde7a (127.7.17.193:34957): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 17db5066d9464d8c93b9ae56567ee14a: Network error: Client connection negotiation failed: client connection to 127.7.17.193:34957: connect: Connection refused (error 111)
I20260927 15:11:19.232081 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 24623
I20260927 15:11:19.237597 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 24261
2026-09-27T15:11:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2496 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T15:11:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:19Z Disabled control of system clock
I20260927 15:11:19.265754 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:39375
--webserver_interface=127.7.17.254
--webserver_port=0
--builtin_ntp_servers=127.7.17.212:36583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:39375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:11:19.377985 24799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:19.378278 24799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:19.378314 24799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:19.382438 24799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:19.382519 24799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:19.382534 24799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:19.382550 24799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:19.382563 24799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:19.387557 24799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:36583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:39375
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:39375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:19.388674 24799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:19.390120 24799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:19.398893 24807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:19.398908 24805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:19.398996 24799 server_base.cc:1034] running on GCE node
W20260927 15:11:19.398893 24804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:19.399813 24799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:19.401007 24799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:19.402173 24799 hybrid_clock.cc:648] HybridClock initialized: now 1790521879402157 us; error 36 us; skew 500 ppm
I20260927 15:11:19.404106 24799 webserver.cc:460] Webserver started at http://127.7.17.254:33899/ using document root <none> and password file <none>
I20260927 15:11:19.404662 24799 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:19.404723 24799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:19.404944 24799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:19.407178 24799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "831ce0a79c914ad0b35229f267cb5990"
format_stamp: "Formatted at 2026-09-27 15:11:19 on dist-test-slave-hhpq"
server_key: "d3264d4d659cb01b98819a210687d870"
server_key_iv: "91cbed74b46187f0a84ca49d86501a21"
server_key_version: "encryptionkey@0"
I20260927 15:11:19.407763 24799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "831ce0a79c914ad0b35229f267cb5990"
format_stamp: "Formatted at 2026-09-27 15:11:19 on dist-test-slave-hhpq"
server_key: "d3264d4d659cb01b98819a210687d870"
server_key_iv: "91cbed74b46187f0a84ca49d86501a21"
server_key_version: "encryptionkey@0"
I20260927 15:11:19.411669 24799 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 15:11:19.414417 24813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:19.415853 24799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:11:19.415973 24799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "831ce0a79c914ad0b35229f267cb5990"
format_stamp: "Formatted at 2026-09-27 15:11:19 on dist-test-slave-hhpq"
server_key: "d3264d4d659cb01b98819a210687d870"
server_key_iv: "91cbed74b46187f0a84ca49d86501a21"
server_key_version: "encryptionkey@0"
I20260927 15:11:19.416107 24799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:19.435137 24799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:19.435920 24799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:19.436110 24799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:19.443392 24799 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:39375
I20260927 15:11:19.443450 24865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:39375 every 8 connection(s)
I20260927 15:11:19.444406 24799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:19.447645 24866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:19.451823 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 24799
I20260927 15:11:19.451933 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:19.452174 7239 external_mini_cluster.cc:1428] Setting key f90c67674fb69a31b2abb00b2cadf25a
I20260927 15:11:19.453573 24866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990: Bootstrap starting.
I20260927 15:11:19.456450 24866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:19.457406 24866 log.cc:824] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:19.459513 24866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990: No bootstrap required, opened a new log
I20260927 15:11:19.462815 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:19.453704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50682 (local address 127.7.17.254:39375)
0927 15:11:19.454384 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:19.454404 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:19.454448 (+ 44us) server_negotiation.cc:407] Connection header received
0927 15:11:19.455325 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:19.455351 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:19.455892 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:19.456290 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:19.456797 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:19.458153 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:19.459074 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:19.459761 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:19.459929 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:19.460662 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:19.460668 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:19.460685 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:19.461223 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:19.461295 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:19.461322 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:19.461421 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:19.461550 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 15:11:19.461834 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":107,"threads_started":1}
I20260927 15:11:19.463896 24866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831ce0a79c914ad0b35229f267cb5990" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39375 } }
I20260927 15:11:19.464135 24866 raft_consensus.cc:374] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:19.464170 24866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 831ce0a79c914ad0b35229f267cb5990, State: Initialized, Role: FOLLOWER
I20260927 15:11:19.464591 24866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831ce0a79c914ad0b35229f267cb5990" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39375 } }
I20260927 15:11:19.464710 24866 raft_consensus.cc:388] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:19.464776 24866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:19.464902 24866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:19.465816 24866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831ce0a79c914ad0b35229f267cb5990" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39375 } }
I20260927 15:11:19.466143 24866 leader_election.cc:302] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 831ce0a79c914ad0b35229f267cb5990; no voters:
I20260927 15:11:19.466425 24866 leader_election.cc:288] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:19.466528 24871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:19.466722 24871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990 [term 1 LEADER]: Becoming Leader. State: Replica: 831ce0a79c914ad0b35229f267cb5990, State: Running, Role: LEADER
I20260927 15:11:19.467021 24871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831ce0a79c914ad0b35229f267cb5990" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39375 } }
I20260927 15:11:19.467597 24866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:19.468628 24872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "831ce0a79c914ad0b35229f267cb5990" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831ce0a79c914ad0b35229f267cb5990" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39375 } } }
I20260927 15:11:19.468760 24872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:19.469049 24873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 831ce0a79c914ad0b35229f267cb5990. Latest consensus state: current_term: 1 leader_uuid: "831ce0a79c914ad0b35229f267cb5990" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831ce0a79c914ad0b35229f267cb5990" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39375 } } }
I20260927 15:11:19.469138 24873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:19.469252 24880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:19.472380 24880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:19.478490 24880 catalog_manager.cc:1347] Generated new cluster ID: 0b5cc998c6794e958fc6779b90bea1d4
I20260927 15:11:19.478564 24880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:19.491932 24880 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:19.493325 24880 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:19.499341 24880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990: Generated new TSK 0
I20260927 15:11:19.500144 24880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:19.512732 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39375
--builtin_ntp_servers=127.7.17.212:36583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:11:19.647114 24890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:19.647406 24890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:19.647447 24890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:19.651520 24890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:19.651587 24890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:19.651643 24890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:11:19.656467 24890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:36583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:19.657615 24890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:19.658965 24890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:19.669168 24895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:19.669106 24896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:19.669267 24890 server_base.cc:1034] running on GCE node
W20260927 15:11:19.669422 24898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:19.669879 24890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:19.670579 24890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:19.671770 24890 hybrid_clock.cc:648] HybridClock initialized: now 1790521879671716 us; error 73 us; skew 500 ppm
I20260927 15:11:19.674091 24890 webserver.cc:460] Webserver started at http://127.7.17.193:36311/ using document root <none> and password file <none>
I20260927 15:11:19.674729 24890 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:19.674808 24890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:19.675051 24890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:19.677397 24890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "8851fb22f9fa4bf8bff10f3706915124"
format_stamp: "Formatted at 2026-09-27 15:11:19 on dist-test-slave-hhpq"
server_key: "af63bd829a3540993adae73b18dcd74b"
server_key_iv: "0b9289be01953a251962fcb55ecba3a9"
server_key_version: "encryptionkey@0"
I20260927 15:11:19.677996 24890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "8851fb22f9fa4bf8bff10f3706915124"
format_stamp: "Formatted at 2026-09-27 15:11:19 on dist-test-slave-hhpq"
server_key: "af63bd829a3540993adae73b18dcd74b"
server_key_iv: "0b9289be01953a251962fcb55ecba3a9"
server_key_version: "encryptionkey@0"
I20260927 15:11:19.682173 24890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 15:11:19.684979 24904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:19.686414 24890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:11:19.686545 24890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "8851fb22f9fa4bf8bff10f3706915124"
format_stamp: "Formatted at 2026-09-27 15:11:19 on dist-test-slave-hhpq"
server_key: "af63bd829a3540993adae73b18dcd74b"
server_key_iv: "0b9289be01953a251962fcb55ecba3a9"
server_key_version: "encryptionkey@0"
I20260927 15:11:19.686671 24890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:19.701179 24890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:19.701910 24890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:19.702090 24890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:19.702723 24890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:19.703707 24890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:19.703755 24890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:19.703809 24890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:19.703837 24890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:19.712304 24890 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:41797
I20260927 15:11:19.712355 25017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:41797 every 8 connection(s)
I20260927 15:11:19.713316 24890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:11:19.715828 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 24890
I20260927 15:11:19.715940 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:11:19.716210 7239 external_mini_cluster.cc:1428] Setting key 854997a8b01f6ab310f0cd1132f6fd61
I20260927 15:11:19.718327 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39375
--builtin_ntp_servers=127.7.17.212:36583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:11:19.723450 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:19.715205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:42095 (local address 127.7.17.254:39375)
0927 15:11:19.715355 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:19.715359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:19.716625 (+ 1266us) server_negotiation.cc:407] Connection header received
0927 15:11:19.718236 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:19.718240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:19.718295 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:19.718392 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:19.719454 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:19.720064 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:19.721455 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:19.721691 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:19.721791 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:19.722949 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:19.722952 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:19.722954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:19.722969 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:19.722990 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:19.722997 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:19.723057 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:19.723222 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 15:11:19.723292 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:11:19.724434 25020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:19.715678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:39375 (local address 127.7.17.193:42095)
0927 15:11:19.716435 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:19.716495 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:19.718006 (+ 1511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:19.718599 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:19.718607 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:19.718708 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:19.719258 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:19.719272 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:19.720206 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:19.720210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:19.721318 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:19.721329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:19.721857 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:19.721862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:19.722027 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:19.722647 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:11:19.722649 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:19.722696 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:11:19.722779 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:19.723068 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:19.723080 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:19.723086 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:11:19.723205 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 15:11:19.723599 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":118,"threads_started":1}
I20260927 15:11:19.725800 25018 heartbeater.cc:347] Connected to a master server at 127.7.17.254:39375
I20260927 15:11:19.726045 25018 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:19.726604 25018 heartbeater.cc:511] Master 127.7.17.254:39375 requested a full tablet report, sending...
I20260927 15:11:19.728113 24830 ts_manager.cc:194] Registered new tserver with Master: 8851fb22f9fa4bf8bff10f3706915124 (127.7.17.193:41797)
I20260927 15:11:19.729338 24830 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.17.193:42095
W20260927 15:11:19.829860 25021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:19.830147 25021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:19.830199 25021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:19.834221 25021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:19.834275 25021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:19.834349 25021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:11:19.839126 25021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:36583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:19.840260 25021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:19.841718 25021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:19.851459 25026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:19.851500 25021 server_base.cc:1034] running on GCE node
W20260927 15:11:19.851459 25029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:19.851502 25027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:19.852135 25021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:19.852890 25021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:19.854079 25021 hybrid_clock.cc:648] HybridClock initialized: now 1790521879854049 us; error 46 us; skew 500 ppm
I20260927 15:11:19.856127 25021 webserver.cc:460] Webserver started at http://127.7.17.194:33415/ using document root <none> and password file <none>
I20260927 15:11:19.856739 25021 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:19.856808 25021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:19.857059 25021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:19.859201 25021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "357d5c4f75e0401bb40dfcf794821ea6"
format_stamp: "Formatted at 2026-09-27 15:11:19 on dist-test-slave-hhpq"
server_key: "3394c8fddb5d62ca1345736f42269768"
server_key_iv: "7a4c451fa1a21ea7b8076493fa582779"
server_key_version: "encryptionkey@0"
I20260927 15:11:19.859781 25021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "357d5c4f75e0401bb40dfcf794821ea6"
format_stamp: "Formatted at 2026-09-27 15:11:19 on dist-test-slave-hhpq"
server_key: "3394c8fddb5d62ca1345736f42269768"
server_key_iv: "7a4c451fa1a21ea7b8076493fa582779"
server_key_version: "encryptionkey@0"
I20260927 15:11:19.863670 25021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:11:19.866302 25035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:19.867643 25021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:11:19.867754 25021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "357d5c4f75e0401bb40dfcf794821ea6"
format_stamp: "Formatted at 2026-09-27 15:11:19 on dist-test-slave-hhpq"
server_key: "3394c8fddb5d62ca1345736f42269768"
server_key_iv: "7a4c451fa1a21ea7b8076493fa582779"
server_key_version: "encryptionkey@0"
I20260927 15:11:19.867864 25021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:19.881024 25021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:19.881734 25021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:19.881909 25021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:19.882557 25021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:19.883522 25021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:19.883579 25021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:19.883618 25021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:19.883646 25021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:19.892230 25021 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:45379
I20260927 15:11:19.892239 25148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:45379 every 8 connection(s)
I20260927 15:11:19.893257 25021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:11:19.894665 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 25021
I20260927 15:11:19.894804 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:11:19.895119 7239 external_mini_cluster.cc:1428] Setting key 19bee2d7f17748e0396f5945680cbd42
I20260927 15:11:19.897351 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39375
--builtin_ntp_servers=127.7.17.212:36583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:11:19.903419 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:19.895234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:34455 (local address 127.7.17.254:39375)
0927 15:11:19.895366 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:19.895371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:19.896778 (+ 1407us) server_negotiation.cc:407] Connection header received
0927 15:11:19.898251 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:19.898255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:19.898310 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:19.898398 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:19.899631 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:19.900164 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:19.901425 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:19.901647 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:19.901738 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:19.902918 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:19.902921 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:19.902923 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:19.902937 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:19.902956 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:19.902965 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:19.903024 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:19.903169 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 15:11:19.903266 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:11:19.904394 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:19.895756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:39375 (local address 127.7.17.194:34455)
0927 15:11:19.896609 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:19.896671 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:19.898032 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:19.898646 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:19.898656 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:19.898772 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:19.899469 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:19.899490 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:19.900308 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:19.900311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:19.901296 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:19.901307 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:19.901807 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:19.901811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:19.901975 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:19.902626 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:11:19.902627 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:19.902676 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:11:19.902760 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:19.903036 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:19.903046 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:19.903051 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:11:19.903165 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 15:11:19.903492 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":138,"threads_started":1}
I20260927 15:11:19.905809 25149 heartbeater.cc:347] Connected to a master server at 127.7.17.254:39375
I20260927 15:11:19.906050 25149 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:19.906594 25149 heartbeater.cc:511] Master 127.7.17.254:39375 requested a full tablet report, sending...
I20260927 15:11:19.907747 24830 ts_manager.cc:194] Registered new tserver with Master: 357d5c4f75e0401bb40dfcf794821ea6 (127.7.17.194:45379)
I20260927 15:11:19.908516 24830 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.17.194:34455
W20260927 15:11:20.011900 25152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:20.012187 25152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:20.012225 25152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:20.016361 25152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:20.016443 25152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:20.016502 25152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:11:20.021522 25152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:36583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:39375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:20.022959 25152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:20.024816 25152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:11:20.034747 25152 server_base.cc:1034] running on GCE node
W20260927 15:11:20.034705 25158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:20.034703 25160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:20.034703 25157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:20.035440 25152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:20.036154 25152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:20.037317 25152 hybrid_clock.cc:648] HybridClock initialized: now 1790521880037288 us; error 44 us; skew 500 ppm
I20260927 15:11:20.039248 25152 webserver.cc:460] Webserver started at http://127.7.17.195:39275/ using document root <none> and password file <none>
I20260927 15:11:20.039822 25152 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:20.039892 25152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:20.040098 25152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:20.042219 25152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "ba2687aff8b5427e89b259ec87ab7911"
format_stamp: "Formatted at 2026-09-27 15:11:20 on dist-test-slave-hhpq"
server_key: "8a9804eb8a7c2a177fba3c9d279051fb"
server_key_iv: "6d429699969cf71632020237368b31f9"
server_key_version: "encryptionkey@0"
I20260927 15:11:20.042771 25152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "ba2687aff8b5427e89b259ec87ab7911"
format_stamp: "Formatted at 2026-09-27 15:11:20 on dist-test-slave-hhpq"
server_key: "8a9804eb8a7c2a177fba3c9d279051fb"
server_key_iv: "6d429699969cf71632020237368b31f9"
server_key_version: "encryptionkey@0"
I20260927 15:11:20.046752 25152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:11:20.049429 25166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:20.050750 25152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:20.050877 25152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "ba2687aff8b5427e89b259ec87ab7911"
format_stamp: "Formatted at 2026-09-27 15:11:20 on dist-test-slave-hhpq"
server_key: "8a9804eb8a7c2a177fba3c9d279051fb"
server_key_iv: "6d429699969cf71632020237368b31f9"
server_key_version: "encryptionkey@0"
I20260927 15:11:20.050992 25152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:20.063970 25152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:20.064736 25152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:20.064963 25152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:20.065663 25152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:20.066674 25152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:20.066726 25152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:20.066781 25152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:20.066807 25152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:20.075848 25152 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:33213
I20260927 15:11:20.075870 25279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:33213 every 8 connection(s)
I20260927 15:11:20.076871 25152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:11:20.083793 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 25152
I20260927 15:11:20.083891 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:11:20.084111 7239 external_mini_cluster.cc:1428] Setting key a0b22ec1a056003d559016b70dba7bd1
I20260927 15:11:20.086575 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:20.078795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:37819 (local address 127.7.17.254:39375)
0927 15:11:20.078920 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:20.078924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:20.080069 (+ 1145us) server_negotiation.cc:407] Connection header received
0927 15:11:20.081597 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:20.081600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:20.081661 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:20.081737 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:20.082676 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.083271 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.084486 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.084780 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.084919 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:20.086024 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:20.086027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:20.086029 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:20.086042 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:20.086069 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:20.086080 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:20.086159 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:20.086385 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 15:11:20.086455 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:11:20.087471 25282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:20.079245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:39375 (local address 127.7.17.195:37819)
0927 15:11:20.079916 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:20.079959 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:20.081358 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:20.081925 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:20.081933 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:20.082023 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:20.082526 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:20.082539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.083392 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.083395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:20.084368 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:20.084378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.084950 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.084954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:20.085139 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:20.085728 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:11:20.085730 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:20.085787 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:11:20.085875 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:20.086171 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:20.086181 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:20.086187 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:11:20.086328 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 15:11:20.086584 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":110,"threads_started":1}
I20260927 15:11:20.088770 25280 heartbeater.cc:347] Connected to a master server at 127.7.17.254:39375
I20260927 15:11:20.089056 25280 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:20.089645 25280 heartbeater.cc:511] Master 127.7.17.254:39375 requested a full tablet report, sending...
I20260927 15:11:20.090775 24830 ts_manager.cc:194] Registered new tserver with Master: ba2687aff8b5427e89b259ec87ab7911 (127.7.17.195:33213)
I20260927 15:11:20.091449 24830 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.17.195:37819
I20260927 15:11:20.098389 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:11:20.104264 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:20.100482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50686 (local address 127.7.17.254:39375)
0927 15:11:20.100633 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:20.100638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:20.100865 (+ 227us) server_negotiation.cc:407] Connection header received
0927 15:11:20.100991 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:20.100994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:20.101044 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:20.101129 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:20.101402 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.101921 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.102779 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.103001 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.103091 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:20.103858 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:20.103862 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:20.103865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:20.103878 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:20.103897 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:20.103904 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:20.103962 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:20.104047 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 15:11:20.104115 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 15:11:20.107514 24830 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:11:20.110639 24830 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:11:20.124881 25293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:20.119158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45414 (local address 127.7.17.193:41797)
0927 15:11:20.119597 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:20.119612 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:20.119641 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:11:20.119716 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:20.119721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:20.119938 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:20.120110 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:20.120584 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.121610 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.123253 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.124091 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.124195 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:20.124398 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:20.124580 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 15:11:20.124691 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":120,"threads_started":1}
I20260927 15:11:20.124871 25294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:20.119069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:45379 (local address 127.0.0.1:57988)
0927 15:11:20.120137 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:20.120150 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:20.120243 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:20.121036 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:20.121039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:20.121058 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:20.121156 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:20.121161 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.122466 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.122469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:20.123395 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:20.123404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.124282 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.124285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:20.124376 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:20.124419 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 15:11:20.124515 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 15:11:20.124665 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"mutex_wait_us":48,"thread_start_us":93,"threads_started":1}
I20260927 15:11:20.124871 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:20.118963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:41797 (local address 127.0.0.1:45414)
0927 15:11:20.119352 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:20.119409 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:20.119547 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:20.120300 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:20.120305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:20.120331 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:20.120472 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:20.120480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.121713 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.121716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:20.123115 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:20.123126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.124219 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.124222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:20.124344 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:20.124416 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 15:11:20.124531 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 15:11:20.124665 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":73,"thread_start_us":106,"threads_started":1}
I20260927 15:11:20.125265 25292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:20.119768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:33213 (local address 127.0.0.1:53496)
0927 15:11:20.120684 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:20.120696 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:20.120769 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:20.121426 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:20.121429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:20.121445 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:20.121523 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:20.121529 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.122491 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.122495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:20.123655 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:20.123664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.124887 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.124891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:20.124978 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:20.125002 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:11:20.125073 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 15:11:20.125144 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":19,"thread_start_us":116,"threads_started":1}
I20260927 15:11:20.125674 25295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:20.119935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53496 (local address 127.7.17.195:33213)
0927 15:11:20.120471 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:20.120490 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:20.120881 (+ 391us) server_negotiation.cc:407] Connection header received
0927 15:11:20.120964 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:20.120968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:20.121157 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:20.121288 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:20.121650 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.122384 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.123785 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.124514 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.124623 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:20.124800 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:20.125439 (+ 639us) server_negotiation.cc:299] Negotiation successful
0927 15:11:20.125544 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":90,"threads_started":1}
I20260927 15:11:20.125859 25296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:20.120035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57988 (local address 127.7.17.194:45379)
0927 15:11:20.120436 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:20.120447 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:20.120468 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:11:20.120541 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:20.120546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:20.120756 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:20.121044 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:20.121271 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.122343 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.123522 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.124162 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.125265 (+ 1103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:20.125572 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:20.125622 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 15:11:20.125714 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":107,"threads_started":1}
I20260927 15:11:20.128316 25214 tablet_service.cc:1467] Processing CreateTablet for tablet 63e64f03e4154fe2a5715f2828758069 (DEFAULT_TABLE table=test-table [id=382a8e0098a047a180d22659ba6879fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:20.128403 25083 tablet_service.cc:1467] Processing CreateTablet for tablet 63e64f03e4154fe2a5715f2828758069 (DEFAULT_TABLE table=test-table [id=382a8e0098a047a180d22659ba6879fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:20.128403 24952 tablet_service.cc:1467] Processing CreateTablet for tablet 63e64f03e4154fe2a5715f2828758069 (DEFAULT_TABLE table=test-table [id=382a8e0098a047a180d22659ba6879fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:20.129336 25214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e64f03e4154fe2a5715f2828758069. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:20.129446 25083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e64f03e4154fe2a5715f2828758069. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:20.129446 24952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e64f03e4154fe2a5715f2828758069. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:20.136147 25297 tablet_bootstrap.cc:492] T 63e64f03e4154fe2a5715f2828758069 P ba2687aff8b5427e89b259ec87ab7911: Bootstrap starting.
I20260927 15:11:20.136685 25298 tablet_bootstrap.cc:492] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124: Bootstrap starting.
I20260927 15:11:20.137128 25299 tablet_bootstrap.cc:492] T 63e64f03e4154fe2a5715f2828758069 P 357d5c4f75e0401bb40dfcf794821ea6: Bootstrap starting.
I20260927 15:11:20.138621 25297 tablet_bootstrap.cc:654] T 63e64f03e4154fe2a5715f2828758069 P ba2687aff8b5427e89b259ec87ab7911: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:20.139295 25299 tablet_bootstrap.cc:654] T 63e64f03e4154fe2a5715f2828758069 P 357d5c4f75e0401bb40dfcf794821ea6: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:20.139511 25297 log.cc:824] T 63e64f03e4154fe2a5715f2828758069 P ba2687aff8b5427e89b259ec87ab7911: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:20.140138 25299 log.cc:824] T 63e64f03e4154fe2a5715f2828758069 P 357d5c4f75e0401bb40dfcf794821ea6: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:20.141837 25297 tablet_bootstrap.cc:492] T 63e64f03e4154fe2a5715f2828758069 P ba2687aff8b5427e89b259ec87ab7911: No bootstrap required, opened a new log
I20260927 15:11:20.142052 25297 ts_tablet_manager.cc:1397] T 63e64f03e4154fe2a5715f2828758069 P ba2687aff8b5427e89b259ec87ab7911: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:11:20.142431 25298 tablet_bootstrap.cc:654] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:20.142716 25299 tablet_bootstrap.cc:492] T 63e64f03e4154fe2a5715f2828758069 P 357d5c4f75e0401bb40dfcf794821ea6: No bootstrap required, opened a new log
I20260927 15:11:20.142918 25299 ts_tablet_manager.cc:1397] T 63e64f03e4154fe2a5715f2828758069 P 357d5c4f75e0401bb40dfcf794821ea6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 15:11:20.143383 25298 log.cc:824] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:20.145731 25297 raft_consensus.cc:348] T 63e64f03e4154fe2a5715f2828758069 P ba2687aff8b5427e89b259ec87ab7911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851fb22f9fa4bf8bff10f3706915124" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41797 } } peers { permanent_uuid: "357d5c4f75e0401bb40dfcf794821ea6" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 45379 } } peers { permanent_uuid: "ba2687aff8b5427e89b259ec87ab7911" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 33213 } }
I20260927 15:11:20.145965 25297 raft_consensus.cc:374] T 63e64f03e4154fe2a5715f2828758069 P ba2687aff8b5427e89b259ec87ab7911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:20.146044 25297 raft_consensus.cc:729] T 63e64f03e4154fe2a5715f2828758069 P ba2687aff8b5427e89b259ec87ab7911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba2687aff8b5427e89b259ec87ab7911, State: Initialized, Role: FOLLOWER
I20260927 15:11:20.146183 25298 tablet_bootstrap.cc:492] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124: No bootstrap required, opened a new log
I20260927 15:11:20.146405 25298 ts_tablet_manager.cc:1397] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260927 15:11:20.146582 25297 consensus_queue.cc:260] T 63e64f03e4154fe2a5715f2828758069 P ba2687aff8b5427e89b259ec87ab7911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851fb22f9fa4bf8bff10f3706915124" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41797 } } peers { permanent_uuid: "357d5c4f75e0401bb40dfcf794821ea6" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 45379 } } peers { permanent_uuid: "ba2687aff8b5427e89b259ec87ab7911" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 33213 } }
I20260927 15:11:20.147289 25280 heartbeater.cc:503] Master 127.7.17.254:39375 was elected leader, sending a full tablet report...
I20260927 15:11:20.147176 25299 raft_consensus.cc:348] T 63e64f03e4154fe2a5715f2828758069 P 357d5c4f75e0401bb40dfcf794821ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851fb22f9fa4bf8bff10f3706915124" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41797 } } peers { permanent_uuid: "357d5c4f75e0401bb40dfcf794821ea6" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 45379 } } peers { permanent_uuid: "ba2687aff8b5427e89b259ec87ab7911" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 33213 } }
I20260927 15:11:20.147449 25299 raft_consensus.cc:374] T 63e64f03e4154fe2a5715f2828758069 P 357d5c4f75e0401bb40dfcf794821ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:20.147504 25299 raft_consensus.cc:729] T 63e64f03e4154fe2a5715f2828758069 P 357d5c4f75e0401bb40dfcf794821ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357d5c4f75e0401bb40dfcf794821ea6, State: Initialized, Role: FOLLOWER
I20260927 15:11:20.147722 25297 ts_tablet_manager.cc:1428] T 63e64f03e4154fe2a5715f2828758069 P ba2687aff8b5427e89b259ec87ab7911: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:11:20.147985 25299 consensus_queue.cc:260] T 63e64f03e4154fe2a5715f2828758069 P 357d5c4f75e0401bb40dfcf794821ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851fb22f9fa4bf8bff10f3706915124" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41797 } } peers { permanent_uuid: "357d5c4f75e0401bb40dfcf794821ea6" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 45379 } } peers { permanent_uuid: "ba2687aff8b5427e89b259ec87ab7911" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 33213 } }
I20260927 15:11:20.149049 25299 ts_tablet_manager.cc:1428] T 63e64f03e4154fe2a5715f2828758069 P 357d5c4f75e0401bb40dfcf794821ea6: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260927 15:11:20.149413 25149 heartbeater.cc:503] Master 127.7.17.254:39375 was elected leader, sending a full tablet report...
I20260927 15:11:20.151167 25298 raft_consensus.cc:348] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851fb22f9fa4bf8bff10f3706915124" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41797 } } peers { permanent_uuid: "357d5c4f75e0401bb40dfcf794821ea6" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 45379 } } peers { permanent_uuid: "ba2687aff8b5427e89b259ec87ab7911" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 33213 } }
I20260927 15:11:20.151466 25298 raft_consensus.cc:374] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:20.151530 25298 raft_consensus.cc:729] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8851fb22f9fa4bf8bff10f3706915124, State: Initialized, Role: FOLLOWER
I20260927 15:11:20.152004 25298 consensus_queue.cc:260] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851fb22f9fa4bf8bff10f3706915124" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41797 } } peers { permanent_uuid: "357d5c4f75e0401bb40dfcf794821ea6" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 45379 } } peers { permanent_uuid: "ba2687aff8b5427e89b259ec87ab7911" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 33213 } }
I20260927 15:11:20.152878 25018 heartbeater.cc:503] Master 127.7.17.254:39375 was elected leader, sending a full tablet report...
I20260927 15:11:20.153075 25298 ts_tablet_manager.cc:1428] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 15:11:20.170274 25305 raft_consensus.cc:482] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:11:20.170486 25305 raft_consensus.cc:504] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851fb22f9fa4bf8bff10f3706915124" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41797 } } peers { permanent_uuid: "357d5c4f75e0401bb40dfcf794821ea6" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 45379 } } peers { permanent_uuid: "ba2687aff8b5427e89b259ec87ab7911" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 33213 } }
I20260927 15:11:20.171667 25305 leader_election.cc:288] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 357d5c4f75e0401bb40dfcf794821ea6 (127.7.17.194:45379), ba2687aff8b5427e89b259ec87ab7911 (127.7.17.195:33213)
I20260927 15:11:20.177170 25296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:20.172004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:44943 (local address 127.7.17.194:45379)
0927 15:11:20.172117 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:20.172121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:20.172666 (+ 545us) server_negotiation.cc:407] Connection header received
0927 15:11:20.172797 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:20.172800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:20.172874 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:20.172958 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:20.173345 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.174137 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.175866 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.176477 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.176598 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:20.176649 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:20.176956 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 15:11:20.177030 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 15:11:20.177170 25295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:20.172346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:38733 (local address 127.7.17.195:33213)
0927 15:11:20.172460 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:20.172463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:20.172635 (+ 172us) server_negotiation.cc:407] Connection header received
0927 15:11:20.172798 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:20.172800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:20.172874 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:20.172958 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:20.173343 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.174129 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.175866 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.176477 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.176598 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:20.176649 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:20.176955 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 15:11:20.177030 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:11:20.177196 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:20.171925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:45379 (local address 127.7.17.193:44943)
0927 15:11:20.172578 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:20.172590 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:20.172699 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:20.172971 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:20.172974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:20.172998 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:20.173232 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:20.173240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.174261 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.174264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:20.175739 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:20.175751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.176614 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.176617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:20.176792 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:20.176863 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 15:11:20.176938 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 15:11:20.177029 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":206,"spinlock_wait_cycles":43520}
I20260927 15:11:20.177199 25020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:20.172226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:33213 (local address 127.7.17.193:38733)
0927 15:11:20.172558 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:20.172574 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:20.172699 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:20.172971 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:20.172974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:20.172998 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:20.173235 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:20.173241 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.174251 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.174255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:20.175739 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:20.175751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.176614 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.176617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:20.176786 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:20.176863 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 15:11:20.176938 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 15:11:20.177029 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":217,"spinlock_wait_cycles":16768,"thread_start_us":110,"threads_started":1}
I20260927 15:11:20.178118 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e64f03e4154fe2a5715f2828758069" candidate_uuid: "8851fb22f9fa4bf8bff10f3706915124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357d5c4f75e0401bb40dfcf794821ea6" is_pre_election: true
I20260927 15:11:20.178289 25234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e64f03e4154fe2a5715f2828758069" candidate_uuid: "8851fb22f9fa4bf8bff10f3706915124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba2687aff8b5427e89b259ec87ab7911" is_pre_election: true
I20260927 15:11:20.178427 25103 raft_consensus.cc:2454] T 63e64f03e4154fe2a5715f2828758069 P 357d5c4f75e0401bb40dfcf794821ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8851fb22f9fa4bf8bff10f3706915124 in term 0.
I20260927 15:11:20.178586 25234 raft_consensus.cc:2454] T 63e64f03e4154fe2a5715f2828758069 P ba2687aff8b5427e89b259ec87ab7911 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8851fb22f9fa4bf8bff10f3706915124 in term 0.
I20260927 15:11:20.178979 24905 leader_election.cc:302] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357d5c4f75e0401bb40dfcf794821ea6, 8851fb22f9fa4bf8bff10f3706915124; no voters:
I20260927 15:11:20.179319 25305 raft_consensus.cc:2781] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:11:20.179414 25305 raft_consensus.cc:482] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:11:20.179458 25305 raft_consensus.cc:3037] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:20.180491 25305 raft_consensus.cc:504] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851fb22f9fa4bf8bff10f3706915124" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41797 } } peers { permanent_uuid: "357d5c4f75e0401bb40dfcf794821ea6" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 45379 } } peers { permanent_uuid: "ba2687aff8b5427e89b259ec87ab7911" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 33213 } }
I20260927 15:11:20.180891 25305 leader_election.cc:288] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124 [CANDIDATE]: Term 1 election: Requested vote from peers 357d5c4f75e0401bb40dfcf794821ea6 (127.7.17.194:45379), ba2687aff8b5427e89b259ec87ab7911 (127.7.17.195:33213)
I20260927 15:11:20.181401 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e64f03e4154fe2a5715f2828758069" candidate_uuid: "8851fb22f9fa4bf8bff10f3706915124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357d5c4f75e0401bb40dfcf794821ea6"
I20260927 15:11:20.181401 25234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e64f03e4154fe2a5715f2828758069" candidate_uuid: "8851fb22f9fa4bf8bff10f3706915124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba2687aff8b5427e89b259ec87ab7911"
I20260927 15:11:20.181550 25103 raft_consensus.cc:3037] T 63e64f03e4154fe2a5715f2828758069 P 357d5c4f75e0401bb40dfcf794821ea6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:20.181550 25234 raft_consensus.cc:3037] T 63e64f03e4154fe2a5715f2828758069 P ba2687aff8b5427e89b259ec87ab7911 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:20.182886 25103 raft_consensus.cc:2454] T 63e64f03e4154fe2a5715f2828758069 P 357d5c4f75e0401bb40dfcf794821ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8851fb22f9fa4bf8bff10f3706915124 in term 1.
I20260927 15:11:20.182886 25234 raft_consensus.cc:2454] T 63e64f03e4154fe2a5715f2828758069 P ba2687aff8b5427e89b259ec87ab7911 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8851fb22f9fa4bf8bff10f3706915124 in term 1.
I20260927 15:11:20.183260 24905 leader_election.cc:302] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8851fb22f9fa4bf8bff10f3706915124, ba2687aff8b5427e89b259ec87ab7911; no voters:
I20260927 15:11:20.183440 25305 raft_consensus.cc:2781] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:20.183674 25305 raft_consensus.cc:686] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124 [term 1 LEADER]: Becoming Leader. State: Replica: 8851fb22f9fa4bf8bff10f3706915124, State: Running, Role: LEADER
I20260927 15:11:20.184036 25305 consensus_queue.cc:237] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851fb22f9fa4bf8bff10f3706915124" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41797 } } peers { permanent_uuid: "357d5c4f75e0401bb40dfcf794821ea6" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 45379 } } peers { permanent_uuid: "ba2687aff8b5427e89b259ec87ab7911" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 33213 } }
I20260927 15:11:20.187369 24830 catalog_manager.cc:5495] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8851fb22f9fa4bf8bff10f3706915124 (127.7.17.193). New cstate: current_term: 1 leader_uuid: "8851fb22f9fa4bf8bff10f3706915124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851fb22f9fa4bf8bff10f3706915124" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 41797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "357d5c4f75e0401bb40dfcf794821ea6" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 45379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba2687aff8b5427e89b259ec87ab7911" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 33213 } health_report { overall_health: UNKNOWN } } }
I20260927 15:11:20.205972 25293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:20.202676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45428 (local address 127.7.17.193:41797)
0927 15:11:20.202784 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:20.202787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:20.202800 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:11:20.202911 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:20.202914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:20.202972 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:20.203069 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:11:20.203261 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.203794 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.204606 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:20.204858 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:20.204942 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:20.205193 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:11:20.205636 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:11:20.205729 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:20.205805 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 15:11:20.205856 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 15:11:20.212181 25103 raft_consensus.cc:1261] T 63e64f03e4154fe2a5715f2828758069 P 357d5c4f75e0401bb40dfcf794821ea6 [term 1 FOLLOWER]: Refusing update from remote peer 8851fb22f9fa4bf8bff10f3706915124: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:20.212218 25234 raft_consensus.cc:1261] T 63e64f03e4154fe2a5715f2828758069 P ba2687aff8b5427e89b259ec87ab7911 [term 1 FOLLOWER]: Refusing update from remote peer 8851fb22f9fa4bf8bff10f3706915124: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:20.212946 25307 consensus_queue.cc:1035] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba2687aff8b5427e89b259ec87ab7911" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:20.213155 25305 consensus_queue.cc:1035] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "357d5c4f75e0401bb40dfcf794821ea6" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 45379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:11:20.214483 25019 tablet.cc:2369] T 63e64f03e4154fe2a5715f2828758069 P 8851fb22f9fa4bf8bff10f3706915124: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:11:20.221884 25311 mvcc.cc:205] Tried to move back new op lower bound from 7333977621341085696 to 7333977621236850688. Current Snapshot: MvccSnapshot[applied={T|T < 7333977621341085696}]
I20260927 15:11:20.223658 25312 mvcc.cc:205] Tried to move back new op lower bound from 7333977621341085696 to 7333977621236850688. Current Snapshot: MvccSnapshot[applied={T|T < 7333977621341085696}]
I20260927 15:11:20.228050 24830 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:20.228250 24830 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:20.230988 24830 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 831ce0a79c914ad0b35229f267cb5990: Sending DeleteTablet for 3 replicas of tablet 63e64f03e4154fe2a5715f2828758069
I20260927 15:11:20.231940 24952 tablet_service.cc:1514] Processing DeleteTablet for tablet 63e64f03e4154fe2a5715f2828758069 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:20 UTC) from {username='slave'} at 127.0.0.1:45414
I20260927 15:11:20.231946 25214 tablet_service.cc:1514] Processing DeleteTablet for tablet 63e64f03e4154fe2a5715f2828758069 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:20 UTC) from {username='slave'} at 127.0.0.1:53496
I20260927 15:11:20.232084 25083 tablet_service.cc:1514] Processing DeleteTablet for tablet 63e64f03e4154fe2a5715f2828758069 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:20 UTC) from {username='slave'} at 127.0.0.1:57988
I20260927 15:11:20.232592 25320 tablet_replica.cc:330] T 63e64f03e4154fe2a5715f2828758069 P ba2687aff8b5427e89b259ec87ab7911: stopping tablet replica
I20260927 15:11:20.232679 25321 tablet_replica.cc:330] T 63e64f03e4154fe2a5715f2828758069 P 357d5c4f75e0401bb40dfcf794821ea6: stopping tablet replica
I20260927 15:11:20.232789 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 24890
I20260927 15:11:20.233000 25321 raft_consensus.cc:2229] T 63e64f03e4154fe2a5715f2828758069 P 357d5c4f75e0401bb40dfcf794821ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:11:20.233422 25320 raft_consensus.cc:2229] T 63e64f03e4154fe2a5715f2828758069 P ba2687aff8b5427e89b259ec87ab7911 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:11:20.233726 25321 raft_consensus.cc:2258] T 63e64f03e4154fe2a5715f2828758069 P 357d5c4f75e0401bb40dfcf794821ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:20.233724 25320 raft_consensus.cc:2258] T 63e64f03e4154fe2a5715f2828758069 P ba2687aff8b5427e89b259ec87ab7911 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:20.235239 25320 ts_tablet_manager.cc:1905] T 63e64f03e4154fe2a5715f2828758069 P ba2687aff8b5427e89b259ec87ab7911: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:20.235936 25321 ts_tablet_manager.cc:1905] T 63e64f03e4154fe2a5715f2828758069 P 357d5c4f75e0401bb40dfcf794821ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:20.237572 25320 ts_tablet_manager.cc:1918] T 63e64f03e4154fe2a5715f2828758069 P ba2687aff8b5427e89b259ec87ab7911: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:20.237666 25320 log.cc:1196] T 63e64f03e4154fe2a5715f2828758069 P ba2687aff8b5427e89b259ec87ab7911: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/63e64f03e4154fe2a5715f2828758069
I20260927 15:11:20.237939 25320 ts_tablet_manager.cc:1939] T 63e64f03e4154fe2a5715f2828758069 P ba2687aff8b5427e89b259ec87ab7911: Deleting consensus metadata
I20260927 15:11:20.238543 25321 ts_tablet_manager.cc:1918] T 63e64f03e4154fe2a5715f2828758069 P 357d5c4f75e0401bb40dfcf794821ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:20.238639 25321 log.cc:1196] T 63e64f03e4154fe2a5715f2828758069 P 357d5c4f75e0401bb40dfcf794821ea6: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521636012660-7239-0/minicluster-data/ts-1/wal/wals/63e64f03e4154fe2a5715f2828758069
I20260927 15:11:20.238978 25321 ts_tablet_manager.cc:1939] T 63e64f03e4154fe2a5715f2828758069 P 357d5c4f75e0401bb40dfcf794821ea6: Deleting consensus metadata
I20260927 15:11:20.239185 24814 catalog_manager.cc:4841] TS ba2687aff8b5427e89b259ec87ab7911 (127.7.17.195:33213): tablet 63e64f03e4154fe2a5715f2828758069 (table test-table [id=382a8e0098a047a180d22659ba6879fc]) successfully deleted
I20260927 15:11:20.240186 24814 catalog_manager.cc:4841] TS 357d5c4f75e0401bb40dfcf794821ea6 (127.7.17.194:45379): tablet 63e64f03e4154fe2a5715f2828758069 (table test-table [id=382a8e0098a047a180d22659ba6879fc]) successfully deleted
W20260927 15:11:20.240614 24815 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.17.193:41797 (error 108)
I20260927 15:11:20.240841 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 25021
I20260927 15:11:20.242697 25292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:20.242414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:41797 (local address 127.0.0.1:45442)
0927 15:11:20.242549 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:20.242619 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.193:41797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 15:11:20.242846 24815 catalog_manager.cc:4584] TS 8851fb22f9fa4bf8bff10f3706915124 (127.7.17.193:41797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63e64f03e4154fe2a5715f2828758069: Network error: Client connection negotiation failed: client connection to 127.7.17.193:41797: connect: Connection refused (error 111)
I20260927 15:11:20.245901 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 25152
I20260927 15:11:20.250566 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 24799
2026-09-27T15:11:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1012 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:20 dist-test-slave-hhpq krb5kdc[25325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:20 dist-test-slave-hhpq krb5kdc[25325](info): set up 2 sockets
Sep 27 15:11:20 dist-test-slave-hhpq krb5kdc[25325](info): commencing operation
krb5kdc: starting...
W20260927 15:11:21.208624 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:11:21 dist-test-slave-hhpq krb5kdc[25325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:11:21.331988 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:43415
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:43415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:21.448067 25338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:21.448366 25338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:21.448424 25338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:21.452510 25338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:21.452592 25338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:21.452618 25338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:21.452639 25338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:21.452661 25338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:21.458189 25338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:43415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:43415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:21.459328 25338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:21.460398 25338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:21.469154 25344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:21.469326 25346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:21.469158 25343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:21.469413 25338 server_base.cc:1034] running on GCE node
I20260927 15:11:21.470165 25338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:21.471354 25338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:21.472517 25338 hybrid_clock.cc:648] HybridClock initialized: now 1790521881472503 us; error 33 us; skew 500 ppm
Sep 27 15:11:21 dist-test-slave-hhpq krb5kdc[25325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521881, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:21.475556 25338 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:11:21.476526 25338 webserver.cc:460] Webserver started at http://127.7.17.254:36265/ using document root <none> and password file <none>
I20260927 15:11:21.477164 25338 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:21.477238 25338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:21.477451 25338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:21.479557 25338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "fbc9ba34c252408c862bdd77737e96a6"
format_stamp: "Formatted at 2026-09-27 15:11:21 on dist-test-slave-hhpq"
server_key: "8e847a9317037a27cdb95b9f81585f42"
server_key_iv: "d4fa460bbd74de54ab77bee3d351a521"
server_key_version: "encryptionkey@0"
I20260927 15:11:21.480140 25338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "fbc9ba34c252408c862bdd77737e96a6"
format_stamp: "Formatted at 2026-09-27 15:11:21 on dist-test-slave-hhpq"
server_key: "8e847a9317037a27cdb95b9f81585f42"
server_key_iv: "d4fa460bbd74de54ab77bee3d351a521"
server_key_version: "encryptionkey@0"
I20260927 15:11:21.484050 25338 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 15:11:21.486774 25353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:21.488255 25338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:11:21.488395 25338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "fbc9ba34c252408c862bdd77737e96a6"
format_stamp: "Formatted at 2026-09-27 15:11:21 on dist-test-slave-hhpq"
server_key: "8e847a9317037a27cdb95b9f81585f42"
server_key_iv: "d4fa460bbd74de54ab77bee3d351a521"
server_key_version: "encryptionkey@0"
I20260927 15:11:21.488513 25338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:21.529009 25338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:21.532366 25338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:21.532565 25338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:21.539762 25338 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:43415
I20260927 15:11:21.539778 25405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:43415 every 8 connection(s)
I20260927 15:11:21.540758 25338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:21.543581 25406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:21.548889 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 25338
I20260927 15:11:21.549036 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:21.549268 25406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6: Bootstrap starting.
I20260927 15:11:21.549335 7239 external_mini_cluster.cc:1428] Setting key a4ae50b93d29500de79371b5ab727568
I20260927 15:11:21.551826 25406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:21.552565 25406 log.cc:824] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:21.554412 25406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6: No bootstrap required, opened a new log
Sep 27 15:11:21 dist-test-slave-hhpq krb5kdc[25325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:11:21.558259 25406 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc9ba34c252408c862bdd77737e96a6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43415 } }
I20260927 15:11:21.558498 25406 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:21.558542 25406 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc9ba34c252408c862bdd77737e96a6, State: Initialized, Role: FOLLOWER
I20260927 15:11:21.559022 25406 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc9ba34c252408c862bdd77737e96a6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43415 } }
I20260927 15:11:21.559202 25406 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:21.559273 25406 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:21.559381 25406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:21.560364 25406 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc9ba34c252408c862bdd77737e96a6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43415 } }
I20260927 15:11:21.560693 25406 leader_election.cc:302] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbc9ba34c252408c862bdd77737e96a6; no voters:
I20260927 15:11:21.561028 25406 leader_election.cc:288] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:21.561123 25411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:21.561390 25411 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6 [term 1 LEADER]: Becoming Leader. State: Replica: fbc9ba34c252408c862bdd77737e96a6, State: Running, Role: LEADER
I20260927 15:11:21.561754 25411 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc9ba34c252408c862bdd77737e96a6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43415 } }
I20260927 15:11:21.562417 25406 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:21.563671 25412 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbc9ba34c252408c862bdd77737e96a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc9ba34c252408c862bdd77737e96a6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43415 } } }
I20260927 15:11:21.563670 25413 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbc9ba34c252408c862bdd77737e96a6. Latest consensus state: current_term: 1 leader_uuid: "fbc9ba34c252408c862bdd77737e96a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc9ba34c252408c862bdd77737e96a6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43415 } } }
I20260927 15:11:21.563822 25413 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:21.563822 25412 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:21.563838 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:21.550741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37722 (local address 127.7.17.254:43415)
0927 15:11:21.551177 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:21.551192 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:21.551230 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:11:21.551923 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:21.551948 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:21.552301 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:21.552620 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:21.553465 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:21.554522 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:21.555386 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:21.555877 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:21.556032 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:21.558728 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:21.558734 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:21.558748 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:21.558781 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:21.560452 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:21.560922 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:21.560927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:21.560932 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:21.561084 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:21.561416 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:21.561419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:21.561421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:21.561954 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:21.562156 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:21.562462 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 15:11:21.562792 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":133,"threads_started":1}
W20260927 15:11:21.567328 25426 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:11:21.567402 25426 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:11:21.567480 25422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:21.568329 25422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:21.574527 25422 catalog_manager.cc:1347] Generated new cluster ID: cfe6e9adbb2745a29149d58d5c8638db
I20260927 15:11:21.574606 25422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:21.594267 25422 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:21.595243 25422 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:21.604626 25422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6: Generated new TSK 0
I20260927 15:11:21.605432 25422 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
I20260927 15:11:21.647694 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43415
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:21.784912 25432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:21.785212 25432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:21.785259 25432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:21.789546 25432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:21.789623 25432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:21.789683 25432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:11:21.795212 25432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:21.796308 25432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:21.797375 25432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:21.807633 25440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:21.807578 25438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:21.807765 25432 server_base.cc:1034] running on GCE node
W20260927 15:11:21.807636 25437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:21.808292 25432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:21.809060 25432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:21.810237 25432 hybrid_clock.cc:648] HybridClock initialized: now 1790521881810210 us; error 43 us; skew 500 ppm
Sep 27 15:11:21 dist-test-slave-hhpq krb5kdc[25325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521881, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:21.816392 25432 init.cc:377] Logged in from keytab as kudu/127.7.17.193@KRBTEST.COM (short username kudu)
I20260927 15:11:21.817436 25432 webserver.cc:460] Webserver started at http://127.7.17.193:44149/ using document root <none> and password file <none>
I20260927 15:11:21.818032 25432 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:21.818102 25432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:21.818303 25432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:21.820449 25432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "9ccf6f8ab5764e459150792b5e5f14dd"
format_stamp: "Formatted at 2026-09-27 15:11:21 on dist-test-slave-hhpq"
server_key: "fa31778f200c22969df9c4f57fad50ce"
server_key_iv: "12df7c7690ad5922bfee8b06d8110ad5"
server_key_version: "encryptionkey@0"
I20260927 15:11:21.821050 25432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "9ccf6f8ab5764e459150792b5e5f14dd"
format_stamp: "Formatted at 2026-09-27 15:11:21 on dist-test-slave-hhpq"
server_key: "fa31778f200c22969df9c4f57fad50ce"
server_key_iv: "12df7c7690ad5922bfee8b06d8110ad5"
server_key_version: "encryptionkey@0"
I20260927 15:11:21.824885 25432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:11:21.827509 25447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:21.828907 25432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:21.829027 25432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "9ccf6f8ab5764e459150792b5e5f14dd"
format_stamp: "Formatted at 2026-09-27 15:11:21 on dist-test-slave-hhpq"
server_key: "fa31778f200c22969df9c4f57fad50ce"
server_key_iv: "12df7c7690ad5922bfee8b06d8110ad5"
server_key_version: "encryptionkey@0"
I20260927 15:11:21.829141 25432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:21.846172 25432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:21.849429 25432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:21.849615 25432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:21.850230 25432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:21.851082 25432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:21.851130 25432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:21.851183 25432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:21.851209 25432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:21.860211 25432 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:35063
I20260927 15:11:21.860258 25560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:35063 every 8 connection(s)
I20260927 15:11:21.861225 25432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:11:21.865236 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 25432
I20260927 15:11:21.865370 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:11:21.865697 7239 external_mini_cluster.cc:1428] Setting key d01b5da50a2608bcb7d3eedf55877ae4
Sep 27 15:11:21 dist-test-slave-hhpq krb5kdc[25325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521881, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.17.194@KRBTEST.COM; defaulting to no policy
I20260927 15:11:21.876513 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:21.863153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:47169 (local address 127.7.17.254:43415)
0927 15:11:21.863292 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:21.863296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:21.864342 (+ 1046us) server_negotiation.cc:407] Connection header received
0927 15:11:21.865958 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:21.865961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:21.866015 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:21.866083 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:21.867622 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:21.868162 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:21.869563 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:21.869848 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:21.869948 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:21.872483 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:21.872488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:21.872491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:21.872526 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:21.874234 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:21.874797 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:21.874801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:21.874803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:21.874892 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:21.875355 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:21.875359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:21.875361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:21.875562 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:21.875644 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:21.876269 (+ 625us) server_negotiation.cc:299] Negotiation successful
0927 15:11:21.876384 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:11:21.877534 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:21.863701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:43415 (local address 127.7.17.193:47169)
0927 15:11:21.864180 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:21.864234 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:21.865674 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:21.866300 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:21.866311 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:21.866756 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:21.867455 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:21.867478 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:21.868295 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:21.868299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:21.869436 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:21.869447 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:21.870000 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:21.870002 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:21.870166 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:21.870780 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:21.870797 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:21.872292 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:21.874367 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:21.874373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:21.874388 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:21.874688 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:21.875038 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:21.875042 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:21.875044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:21.875247 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:21.875667 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:21.875673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:21.875946 (+ 273us) client_negotiation.cc:769] Sending connection context
0927 15:11:21.876208 (+ 262us) client_negotiation.cc:240] Negotiation successful
0927 15:11:21.876501 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":120,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.194@KRBTEST.COM" created.
I20260927 15:11:21.879009 25561 heartbeater.cc:347] Connected to a master server at 127.7.17.254:43415
I20260927 15:11:21.879313 25561 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:21.879863 25561 heartbeater.cc:511] Master 127.7.17.254:43415 requested a full tablet report, sending...
I20260927 15:11:21.881654 25370 ts_manager.cc:194] Registered new tserver with Master: 9ccf6f8ab5764e459150792b5e5f14dd (127.7.17.193:35063)
I20260927 15:11:21.883354 25370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:47169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
I20260927 15:11:21.896576 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43415
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:22.008659 25566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:22.008980 25566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:22.009033 25566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:22.013093 25566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:22.013161 25566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:22.013222 25566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:11:22.018682 25566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:22.019729 25566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:22.020690 25566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:22.030652 25572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:22.030655 25571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:22.030683 25574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:22.031281 25566 server_base.cc:1034] running on GCE node
I20260927 15:11:22.031804 25566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:22.032593 25566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:22.033757 25566 hybrid_clock.cc:648] HybridClock initialized: now 1790521882033715 us; error 55 us; skew 500 ppm
Sep 27 15:11:22 dist-test-slave-hhpq krb5kdc[25325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521882, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:22.036497 25566 init.cc:377] Logged in from keytab as kudu/127.7.17.194@KRBTEST.COM (short username kudu)
I20260927 15:11:22.037489 25566 webserver.cc:460] Webserver started at http://127.7.17.194:43037/ using document root <none> and password file <none>
I20260927 15:11:22.038033 25566 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:22.038090 25566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:22.038261 25566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:22.040349 25566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "1e0717d64bd54fffb6880899efbf0b0b"
format_stamp: "Formatted at 2026-09-27 15:11:22 on dist-test-slave-hhpq"
server_key: "bc01a878a20ad758e1244f51141ea7c0"
server_key_iv: "d786c81abaadee9caf19bdb895c135cb"
server_key_version: "encryptionkey@0"
I20260927 15:11:22.040936 25566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "1e0717d64bd54fffb6880899efbf0b0b"
format_stamp: "Formatted at 2026-09-27 15:11:22 on dist-test-slave-hhpq"
server_key: "bc01a878a20ad758e1244f51141ea7c0"
server_key_iv: "d786c81abaadee9caf19bdb895c135cb"
server_key_version: "encryptionkey@0"
I20260927 15:11:22.044929 25566 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:11:22.047575 25581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:22.049031 25566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:22.049160 25566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "1e0717d64bd54fffb6880899efbf0b0b"
format_stamp: "Formatted at 2026-09-27 15:11:22 on dist-test-slave-hhpq"
server_key: "bc01a878a20ad758e1244f51141ea7c0"
server_key_iv: "d786c81abaadee9caf19bdb895c135cb"
server_key_version: "encryptionkey@0"
I20260927 15:11:22.049275 25566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:22.064287 25566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:22.067552 25566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:22.067746 25566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:22.068348 25566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:22.069269 25566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:22.069319 25566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:22.069371 25566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:22.069397 25566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:22.078424 25566 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:36475
I20260927 15:11:22.078444 25694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:36475 every 8 connection(s)
I20260927 15:11:22.079375 25566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:11:22.083226 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 25566
I20260927 15:11:22.083343 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:11:22.083636 7239 external_mini_cluster.cc:1428] Setting key 962b82528820fd72cb0e657b3e348dea
Sep 27 15:11:22 dist-test-slave-hhpq krb5kdc[25325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521882, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.17.195@KRBTEST.COM; defaulting to no policy
I20260927 15:11:22.094056 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:22.081284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:36057 (local address 127.7.17.254:43415)
0927 15:11:22.081421 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:22.081425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:22.082316 (+ 891us) server_negotiation.cc:407] Connection header received
0927 15:11:22.083246 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:22.083250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:22.083309 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:22.083391 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:22.084871 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.085486 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.087051 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.087273 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.087357 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:22.090177 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:22.090180 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:22.090182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:22.090207 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:22.091640 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:22.092287 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:22.092290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:22.092291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:22.092371 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:22.092779 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:22.092782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:22.092783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:22.093081 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:22.093198 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:22.093748 (+ 550us) server_negotiation.cc:299] Negotiation successful
0927 15:11:22.093891 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:11:22.095374 25697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:22.081675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:43415 (local address 127.7.17.194:36057)
0927 15:11:22.082155 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:22.082200 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:22.083034 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:22.083619 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:22.083639 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:22.084131 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:22.084721 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:22.084736 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.085615 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.085619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:22.086919 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:22.086932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.087412 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.087417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:22.087601 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:22.088187 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:22.088237 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:22.089982 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:22.091761 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:22.091768 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:22.091785 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:22.092178 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:22.092483 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:22.092486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:22.092488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:22.092677 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:22.093196 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:22.093204 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:22.093493 (+ 289us) client_negotiation.cc:769] Sending connection context
0927 15:11:22.093687 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 15:11:22.094108 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.195@KRBTEST.COM" created.
I20260927 15:11:22.096616 25695 heartbeater.cc:347] Connected to a master server at 127.7.17.254:43415
I20260927 15:11:22.096928 25695 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:22.097568 25695 heartbeater.cc:511] Master 127.7.17.254:43415 requested a full tablet report, sending...
I20260927 15:11:22.098760 25370 ts_manager.cc:194] Registered new tserver with Master: 1e0717d64bd54fffb6880899efbf0b0b (127.7.17.194:36475)
I20260927 15:11:22.099364 25370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:36057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
I20260927 15:11:22.112612 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43415
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:22.224269 25700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:22.224566 25700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:22.224628 25700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:22.228752 25700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:22.228853 25700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:22.228935 25700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:11:22.234512 25700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:22.235653 25700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:22.236742 25700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:22.246575 25706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:22.246598 25705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:22.246784 25700 server_base.cc:1034] running on GCE node
W20260927 15:11:22.246600 25708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:22.247262 25700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:22.247990 25700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:22.249145 25700 hybrid_clock.cc:648] HybridClock initialized: now 1790521882249126 us; error 32 us; skew 500 ppm
Sep 27 15:11:22 dist-test-slave-hhpq krb5kdc[25325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521882, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:22.251746 25700 init.cc:377] Logged in from keytab as kudu/127.7.17.195@KRBTEST.COM (short username kudu)
I20260927 15:11:22.252741 25700 webserver.cc:460] Webserver started at http://127.7.17.195:45773/ using document root <none> and password file <none>
I20260927 15:11:22.253315 25700 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:22.253384 25700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:22.253599 25700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:22.255703 25700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "07067b0671714947a205df92a8070061"
format_stamp: "Formatted at 2026-09-27 15:11:22 on dist-test-slave-hhpq"
server_key: "21a18a8ce7e37880b6a488d60e8aa744"
server_key_iv: "0b27415b2607baa8b4f14f77dd6f5ffd"
server_key_version: "encryptionkey@0"
I20260927 15:11:22.256238 25700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "07067b0671714947a205df92a8070061"
format_stamp: "Formatted at 2026-09-27 15:11:22 on dist-test-slave-hhpq"
server_key: "21a18a8ce7e37880b6a488d60e8aa744"
server_key_iv: "0b27415b2607baa8b4f14f77dd6f5ffd"
server_key_version: "encryptionkey@0"
I20260927 15:11:22.260162 25700 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:11:22.262976 25715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:22.264408 25700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 15:11:22.264544 25700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "07067b0671714947a205df92a8070061"
format_stamp: "Formatted at 2026-09-27 15:11:22 on dist-test-slave-hhpq"
server_key: "21a18a8ce7e37880b6a488d60e8aa744"
server_key_iv: "0b27415b2607baa8b4f14f77dd6f5ffd"
server_key_version: "encryptionkey@0"
I20260927 15:11:22.264689 25700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:22.292095 25700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:22.295607 25700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:22.295814 25700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:22.296445 25700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:22.297353 25700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:22.297402 25700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:22.297456 25700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:22.297485 25700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:22.306363 25700 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:38393
I20260927 15:11:22.306398 25828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:38393 every 8 connection(s)
I20260927 15:11:22.307314 25700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:11:22.309124 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 25700
I20260927 15:11:22.309247 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:11:22.309554 7239 external_mini_cluster.cc:1428] Setting key 0b8ba0a6cdc952aa9c8ea2fc24a08d6e
Sep 27 15:11:22 dist-test-slave-hhpq krb5kdc[25325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521882, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:11:22.320595 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:22.309273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:32883 (local address 127.7.17.254:43415)
0927 15:11:22.309427 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:22.309432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:22.310336 (+ 904us) server_negotiation.cc:407] Connection header received
0927 15:11:22.311397 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:22.311400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:22.311451 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:22.311532 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:22.312807 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.313426 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.314600 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.314825 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.314906 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:22.317356 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:22.317359 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:22.317361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:22.317388 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:22.318717 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:22.319221 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:22.319224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:22.319225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:22.319312 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:22.319595 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:22.319598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:22.319599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:22.319779 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:22.319852 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:22.320361 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 15:11:22.320467 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:11:22.321552 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:22.309628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:43415 (local address 127.7.17.195:32883)
0927 15:11:22.310184 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:22.310233 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:22.311152 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:22.311738 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:22.311747 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:22.312157 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:22.312656 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:22.312669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.313566 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.313569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:22.314487 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:22.314496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.314946 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.314948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:22.315094 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:22.315631 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:22.315646 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:22.317181 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:22.318826 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:22.318831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:22.318843 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:22.319113 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:22.319392 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:22.319394 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:22.319396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:22.319527 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:22.319866 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:22.319871 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:22.320128 (+ 257us) client_negotiation.cc:769] Sending connection context
0927 15:11:22.320288 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 15:11:22.320564 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":118,"threads_started":1}
I20260927 15:11:22.322783 25829 heartbeater.cc:347] Connected to a master server at 127.7.17.254:43415
I20260927 15:11:22.323050 25829 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:22.323637 25829 heartbeater.cc:511] Master 127.7.17.254:43415 requested a full tablet report, sending...
I20260927 15:11:22.324673 25370 ts_manager.cc:194] Registered new tserver with Master: 07067b0671714947a205df92a8070061 (127.7.17.195:38393)
I20260927 15:11:22.325271 25370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:32883
I20260927 15:11:22.335566 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:11:22.344981 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:22.337515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37726 (local address 127.7.17.254:43415)
0927 15:11:22.337635 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:22.337640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:22.337781 (+ 141us) server_negotiation.cc:407] Connection header received
0927 15:11:22.337889 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:22.337891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:22.337930 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:22.337994 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:22.338649 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.339148 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.339943 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.340148 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.340232 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:22.341835 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:22.341839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:22.341841 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:22.341867 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:22.343193 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:22.343694 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:22.343697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:22.343698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:22.343779 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:22.344071 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:22.344074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:22.344076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:22.344302 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:22.344425 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:22.344730 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 15:11:22.344817 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:11:22.348138 25370 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:11:22.351230 25370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:11:22.365289 25841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:22.359701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39976 (local address 127.7.17.195:38393)
0927 15:11:22.360029 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:22.360040 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:22.360086 (+ 46us) server_negotiation.cc:407] Connection header received
0927 15:11:22.360291 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:22.360298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:22.360527 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:22.361091 (+ 564us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:22.361253 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.362158 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.363755 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.364605 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.364713 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:22.364895 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:22.365049 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 15:11:22.365145 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":118,"threads_started":1}
I20260927 15:11:22.365280 25840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:22.359587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:38393 (local address 127.0.0.1:39976)
0927 15:11:22.359960 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:22.360014 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:22.360138 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:22.360729 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:22.360736 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:22.360757 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:22.360933 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:22.360941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.362329 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.362332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:22.363651 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:22.363662 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.364723 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.364726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:22.364860 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:22.364898 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 15:11:22.365005 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 15:11:22.365129 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260927 15:11:22.365906 25842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:22.360357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:36475 (local address 127.0.0.1:39268)
0927 15:11:22.360611 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:22.360624 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:22.360702 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:22.361428 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:22.361431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:22.361449 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:22.361541 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:22.361548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.363304 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.363307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:22.364425 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:22.364436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.365547 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.365550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:22.365638 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:22.365659 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:11:22.365707 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 15:11:22.365773 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":2,"thread_start_us":110,"threads_started":1}
I20260927 15:11:22.366108 25844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:22.360688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:35063 (local address 127.0.0.1:46760)
0927 15:11:22.361436 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:22.361448 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:22.361531 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:22.362077 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:22.362079 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:22.362096 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:22.362177 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:22.362183 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.363187 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.363191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:22.364426 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:22.364436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.365734 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.365738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:22.365836 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:22.365856 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 15:11:22.365924 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 15:11:22.365986 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260927 15:11:22.366256 25845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:22.361018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46760 (local address 127.7.17.193:35063)
0927 15:11:22.361636 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:22.361652 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:22.361673 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:11:22.361746 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:22.361752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:22.361942 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:22.362087 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:22.362319 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.363070 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.364571 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.365415 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.365526 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:22.365679 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:22.366003 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 15:11:22.366107 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":117,"threads_started":1}
I20260927 15:11:22.366750 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:22.360508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39268 (local address 127.7.17.194:36475)
0927 15:11:22.360951 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:22.360963 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:22.360986 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:11:22.361062 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:22.361067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:22.361278 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:22.361705 (+ 427us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:22.361856 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.363171 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.364565 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.365415 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.366338 (+ 923us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:22.366472 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:22.366520 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 15:11:22.366609 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":103,"threads_started":1}
I20260927 15:11:22.368760 25763 tablet_service.cc:1467] Processing CreateTablet for tablet 67f7078192f043e3a6f238d133273386 (DEFAULT_TABLE table=test-table [id=2c18a4b8bee542b8a9926ba7d7a99516]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:22.369287 25495 tablet_service.cc:1467] Processing CreateTablet for tablet 67f7078192f043e3a6f238d133273386 (DEFAULT_TABLE table=test-table [id=2c18a4b8bee542b8a9926ba7d7a99516]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:22.369287 25629 tablet_service.cc:1467] Processing CreateTablet for tablet 67f7078192f043e3a6f238d133273386 (DEFAULT_TABLE table=test-table [id=2c18a4b8bee542b8a9926ba7d7a99516]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:22.369877 25763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67f7078192f043e3a6f238d133273386. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:22.370323 25629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67f7078192f043e3a6f238d133273386. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:22.370323 25495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67f7078192f043e3a6f238d133273386. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:22.377556 25847 tablet_bootstrap.cc:492] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd: Bootstrap starting.
I20260927 15:11:22.377833 25848 tablet_bootstrap.cc:492] T 67f7078192f043e3a6f238d133273386 P 1e0717d64bd54fffb6880899efbf0b0b: Bootstrap starting.
I20260927 15:11:22.379851 25847 tablet_bootstrap.cc:654] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:22.380261 25848 tablet_bootstrap.cc:654] T 67f7078192f043e3a6f238d133273386 P 1e0717d64bd54fffb6880899efbf0b0b: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:22.380681 25847 log.cc:824] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:22.380951 25848 log.cc:824] T 67f7078192f043e3a6f238d133273386 P 1e0717d64bd54fffb6880899efbf0b0b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:22.381192 25846 tablet_bootstrap.cc:492] T 67f7078192f043e3a6f238d133273386 P 07067b0671714947a205df92a8070061: Bootstrap starting.
I20260927 15:11:22.382819 25847 tablet_bootstrap.cc:492] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd: No bootstrap required, opened a new log
I20260927 15:11:22.382822 25848 tablet_bootstrap.cc:492] T 67f7078192f043e3a6f238d133273386 P 1e0717d64bd54fffb6880899efbf0b0b: No bootstrap required, opened a new log
I20260927 15:11:22.383016 25848 ts_tablet_manager.cc:1397] T 67f7078192f043e3a6f238d133273386 P 1e0717d64bd54fffb6880899efbf0b0b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 15:11:22.383059 25847 ts_tablet_manager.cc:1397] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:11:22.383383 25846 tablet_bootstrap.cc:654] T 67f7078192f043e3a6f238d133273386 P 07067b0671714947a205df92a8070061: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:22.384624 25846 log.cc:824] T 67f7078192f043e3a6f238d133273386 P 07067b0671714947a205df92a8070061: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:22.386200 25848 raft_consensus.cc:348] T 67f7078192f043e3a6f238d133273386 P 1e0717d64bd54fffb6880899efbf0b0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07067b0671714947a205df92a8070061" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38393 } } peers { permanent_uuid: "1e0717d64bd54fffb6880899efbf0b0b" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 36475 } } peers { permanent_uuid: "9ccf6f8ab5764e459150792b5e5f14dd" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 35063 } }
I20260927 15:11:22.386437 25848 raft_consensus.cc:374] T 67f7078192f043e3a6f238d133273386 P 1e0717d64bd54fffb6880899efbf0b0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:22.386515 25848 raft_consensus.cc:729] T 67f7078192f043e3a6f238d133273386 P 1e0717d64bd54fffb6880899efbf0b0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e0717d64bd54fffb6880899efbf0b0b, State: Initialized, Role: FOLLOWER
I20260927 15:11:22.386857 25846 tablet_bootstrap.cc:492] T 67f7078192f043e3a6f238d133273386 P 07067b0671714947a205df92a8070061: No bootstrap required, opened a new log
I20260927 15:11:22.386984 25848 consensus_queue.cc:260] T 67f7078192f043e3a6f238d133273386 P 1e0717d64bd54fffb6880899efbf0b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07067b0671714947a205df92a8070061" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38393 } } peers { permanent_uuid: "1e0717d64bd54fffb6880899efbf0b0b" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 36475 } } peers { permanent_uuid: "9ccf6f8ab5764e459150792b5e5f14dd" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 35063 } }
I20260927 15:11:22.387077 25846 ts_tablet_manager.cc:1397] T 67f7078192f043e3a6f238d133273386 P 07067b0671714947a205df92a8070061: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:11:22.387180 25847 raft_consensus.cc:348] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07067b0671714947a205df92a8070061" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38393 } } peers { permanent_uuid: "1e0717d64bd54fffb6880899efbf0b0b" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 36475 } } peers { permanent_uuid: "9ccf6f8ab5764e459150792b5e5f14dd" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 35063 } }
I20260927 15:11:22.387655 25695 heartbeater.cc:503] Master 127.7.17.254:43415 was elected leader, sending a full tablet report...
I20260927 15:11:22.388036 25848 ts_tablet_manager.cc:1428] T 67f7078192f043e3a6f238d133273386 P 1e0717d64bd54fffb6880899efbf0b0b: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260927 15:11:22.388669 25847 raft_consensus.cc:374] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:22.388749 25847 raft_consensus.cc:729] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ccf6f8ab5764e459150792b5e5f14dd, State: Initialized, Role: FOLLOWER
I20260927 15:11:22.389321 25847 consensus_queue.cc:260] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07067b0671714947a205df92a8070061" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38393 } } peers { permanent_uuid: "1e0717d64bd54fffb6880899efbf0b0b" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 36475 } } peers { permanent_uuid: "9ccf6f8ab5764e459150792b5e5f14dd" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 35063 } }
I20260927 15:11:22.390412 25847 ts_tablet_manager.cc:1428] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 15:11:22.390784 25561 heartbeater.cc:503] Master 127.7.17.254:43415 was elected leader, sending a full tablet report...
I20260927 15:11:22.391047 25846 raft_consensus.cc:348] T 67f7078192f043e3a6f238d133273386 P 07067b0671714947a205df92a8070061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07067b0671714947a205df92a8070061" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38393 } } peers { permanent_uuid: "1e0717d64bd54fffb6880899efbf0b0b" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 36475 } } peers { permanent_uuid: "9ccf6f8ab5764e459150792b5e5f14dd" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 35063 } }
I20260927 15:11:22.391316 25846 raft_consensus.cc:374] T 67f7078192f043e3a6f238d133273386 P 07067b0671714947a205df92a8070061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:22.391374 25846 raft_consensus.cc:729] T 67f7078192f043e3a6f238d133273386 P 07067b0671714947a205df92a8070061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07067b0671714947a205df92a8070061, State: Initialized, Role: FOLLOWER
I20260927 15:11:22.391829 25846 consensus_queue.cc:260] T 67f7078192f043e3a6f238d133273386 P 07067b0671714947a205df92a8070061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07067b0671714947a205df92a8070061" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38393 } } peers { permanent_uuid: "1e0717d64bd54fffb6880899efbf0b0b" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 36475 } } peers { permanent_uuid: "9ccf6f8ab5764e459150792b5e5f14dd" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 35063 } }
I20260927 15:11:22.392621 25829 heartbeater.cc:503] Master 127.7.17.254:43415 was elected leader, sending a full tablet report...
I20260927 15:11:22.392895 25846 ts_tablet_manager.cc:1428] T 67f7078192f043e3a6f238d133273386 P 07067b0671714947a205df92a8070061: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 15:11:22.492041 25853 raft_consensus.cc:482] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:11:22.492280 25853 raft_consensus.cc:504] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07067b0671714947a205df92a8070061" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38393 } } peers { permanent_uuid: "1e0717d64bd54fffb6880899efbf0b0b" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 36475 } } peers { permanent_uuid: "9ccf6f8ab5764e459150792b5e5f14dd" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 35063 } }
I20260927 15:11:22.493531 25853 leader_election.cc:288] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07067b0671714947a205df92a8070061 (127.7.17.195:38393), 1e0717d64bd54fffb6880899efbf0b0b (127.7.17.194:36475)
I20260927 15:11:22.498569 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:22.494353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:55959 (local address 127.7.17.194:36475)
0927 15:11:22.494479 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:22.494484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:22.494640 (+ 156us) server_negotiation.cc:407] Connection header received
0927 15:11:22.494744 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:22.494746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:22.494787 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:22.494857 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:22.495135 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.495820 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.497294 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.497916 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.498089 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:22.498133 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:22.498363 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 15:11:22.498437 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:11:22.498569 25841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:22.493892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:39543 (local address 127.7.17.195:38393)
0927 15:11:22.494043 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:22.494048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:22.494247 (+ 199us) server_negotiation.cc:407] Connection header received
0927 15:11:22.494400 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:22.494403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:22.494462 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:22.494537 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:22.494868 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.495627 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.497285 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.497916 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.498047 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:22.498105 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:22.498363 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 15:11:22.498437 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:11:22.498618 25855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:22.493782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:38393 (local address 127.7.17.193:39543)
0927 15:11:22.494124 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:22.494137 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:22.494276 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:22.494565 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:22.494568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:22.494601 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:22.494762 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:22.494768 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.495741 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.495744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:22.497120 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:22.497133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.498056 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.498060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:22.498225 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:22.498270 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 15:11:22.498336 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 15:11:22.498431 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":101,"spinlock_wait_cycles":10112,"thread_start_us":129,"threads_started":1}
I20260927 15:11:22.498622 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:22.494195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:36475 (local address 127.7.17.193:55959)
0927 15:11:22.494536 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:22.494548 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:22.494662 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:22.494886 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:22.494890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:22.494906 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:22.494995 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:22.494999 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.495923 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.495926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:22.497178 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:22.497185 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.498056 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.498060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:22.498210 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:22.498262 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 15:11:22.498336 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 15:11:22.498431 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":64,"thread_start_us":118,"threads_started":1}
I20260927 15:11:22.499404 25649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67f7078192f043e3a6f238d133273386" candidate_uuid: "9ccf6f8ab5764e459150792b5e5f14dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e0717d64bd54fffb6880899efbf0b0b" is_pre_election: true
I20260927 15:11:22.499408 25783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67f7078192f043e3a6f238d133273386" candidate_uuid: "9ccf6f8ab5764e459150792b5e5f14dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07067b0671714947a205df92a8070061" is_pre_election: true
I20260927 15:11:22.499748 25649 raft_consensus.cc:2454] T 67f7078192f043e3a6f238d133273386 P 1e0717d64bd54fffb6880899efbf0b0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ccf6f8ab5764e459150792b5e5f14dd in term 0.
I20260927 15:11:22.499748 25783 raft_consensus.cc:2454] T 67f7078192f043e3a6f238d133273386 P 07067b0671714947a205df92a8070061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ccf6f8ab5764e459150792b5e5f14dd in term 0.
I20260927 15:11:22.500277 25450 leader_election.cc:302] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07067b0671714947a205df92a8070061, 9ccf6f8ab5764e459150792b5e5f14dd; no voters:
I20260927 15:11:22.500550 25853 raft_consensus.cc:2781] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:11:22.500643 25853 raft_consensus.cc:482] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:11:22.500685 25853 raft_consensus.cc:3037] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:22.501873 25853 raft_consensus.cc:504] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07067b0671714947a205df92a8070061" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38393 } } peers { permanent_uuid: "1e0717d64bd54fffb6880899efbf0b0b" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 36475 } } peers { permanent_uuid: "9ccf6f8ab5764e459150792b5e5f14dd" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 35063 } }
I20260927 15:11:22.502233 25853 leader_election.cc:288] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd [CANDIDATE]: Term 1 election: Requested vote from peers 07067b0671714947a205df92a8070061 (127.7.17.195:38393), 1e0717d64bd54fffb6880899efbf0b0b (127.7.17.194:36475)
I20260927 15:11:22.502606 25783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67f7078192f043e3a6f238d133273386" candidate_uuid: "9ccf6f8ab5764e459150792b5e5f14dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07067b0671714947a205df92a8070061"
I20260927 15:11:22.502660 25649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67f7078192f043e3a6f238d133273386" candidate_uuid: "9ccf6f8ab5764e459150792b5e5f14dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e0717d64bd54fffb6880899efbf0b0b"
I20260927 15:11:22.502743 25783 raft_consensus.cc:3037] T 67f7078192f043e3a6f238d133273386 P 07067b0671714947a205df92a8070061 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:22.502769 25649 raft_consensus.cc:3037] T 67f7078192f043e3a6f238d133273386 P 1e0717d64bd54fffb6880899efbf0b0b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:22.503996 25649 raft_consensus.cc:2454] T 67f7078192f043e3a6f238d133273386 P 1e0717d64bd54fffb6880899efbf0b0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ccf6f8ab5764e459150792b5e5f14dd in term 1.
I20260927 15:11:22.503996 25783 raft_consensus.cc:2454] T 67f7078192f043e3a6f238d133273386 P 07067b0671714947a205df92a8070061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ccf6f8ab5764e459150792b5e5f14dd in term 1.
I20260927 15:11:22.504299 25450 leader_election.cc:302] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e0717d64bd54fffb6880899efbf0b0b, 9ccf6f8ab5764e459150792b5e5f14dd; no voters:
I20260927 15:11:22.504463 25853 raft_consensus.cc:2781] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:22.504691 25853 raft_consensus.cc:686] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd [term 1 LEADER]: Becoming Leader. State: Replica: 9ccf6f8ab5764e459150792b5e5f14dd, State: Running, Role: LEADER
I20260927 15:11:22.505040 25853 consensus_queue.cc:237] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07067b0671714947a205df92a8070061" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38393 } } peers { permanent_uuid: "1e0717d64bd54fffb6880899efbf0b0b" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 36475 } } peers { permanent_uuid: "9ccf6f8ab5764e459150792b5e5f14dd" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 35063 } }
I20260927 15:11:22.508558 25369 catalog_manager.cc:5495] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ccf6f8ab5764e459150792b5e5f14dd (127.7.17.193). New cstate: current_term: 1 leader_uuid: "9ccf6f8ab5764e459150792b5e5f14dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07067b0671714947a205df92a8070061" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e0717d64bd54fffb6880899efbf0b0b" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 36475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ccf6f8ab5764e459150792b5e5f14dd" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 35063 } health_report { overall_health: HEALTHY } } }
I20260927 15:11:22.519968 25845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:22.516506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46766 (local address 127.7.17.193:35063)
0927 15:11:22.516636 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:22.516641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:22.516655 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:11:22.516780 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:22.516783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:22.516869 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:22.516972 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:11:22.517202 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.517699 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.518561 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.518771 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.518856 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:22.519163 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:11:22.519565 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:11:22.519695 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:22.519787 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 15:11:22.519826 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:11:22.525949 25783 raft_consensus.cc:1261] T 67f7078192f043e3a6f238d133273386 P 07067b0671714947a205df92a8070061 [term 1 FOLLOWER]: Refusing update from remote peer 9ccf6f8ab5764e459150792b5e5f14dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:22.525964 25649 raft_consensus.cc:1261] T 67f7078192f043e3a6f238d133273386 P 1e0717d64bd54fffb6880899efbf0b0b [term 1 FOLLOWER]: Refusing update from remote peer 9ccf6f8ab5764e459150792b5e5f14dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:22.526680 25857 consensus_queue.cc:1035] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "07067b0671714947a205df92a8070061" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 38393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:22.526880 25853 consensus_queue.cc:1035] T 67f7078192f043e3a6f238d133273386 P 9ccf6f8ab5764e459150792b5e5f14dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e0717d64bd54fffb6880899efbf0b0b" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 36475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:22.546214 25368 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:22.546380 25368 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:22.549008 25368 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fbc9ba34c252408c862bdd77737e96a6: Sending DeleteTablet for 3 replicas of tablet 67f7078192f043e3a6f238d133273386
I20260927 15:11:22.550016 25629 tablet_service.cc:1514] Processing DeleteTablet for tablet 67f7078192f043e3a6f238d133273386 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:22 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:39268
I20260927 15:11:22.550179 25495 tablet_service.cc:1514] Processing DeleteTablet for tablet 67f7078192f043e3a6f238d133273386 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:22 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:46760
I20260927 15:11:22.550784 25763 tablet_service.cc:1514] Processing DeleteTablet for tablet 67f7078192f043e3a6f238d133273386 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:22 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:39976
I20260927 15:11:22.551363 25873 tablet_replica.cc:330] T 67f7078192f043e3a6f238d133273386 P 07067b0671714947a205df92a8070061: stopping tablet replica
I20260927 15:11:22.551443 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 25432
I20260927 15:11:22.551751 25873 raft_consensus.cc:2229] T 67f7078192f043e3a6f238d133273386 P 07067b0671714947a205df92a8070061 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:11:22.552203 25873 raft_consensus.cc:2258] T 67f7078192f043e3a6f238d133273386 P 07067b0671714947a205df92a8070061 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:22.554257 25873 ts_tablet_manager.cc:1905] T 67f7078192f043e3a6f238d133273386 P 07067b0671714947a205df92a8070061: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:22.555158 25871 tablet_replica.cc:330] T 67f7078192f043e3a6f238d133273386 P 1e0717d64bd54fffb6880899efbf0b0b: stopping tablet replica
I20260927 15:11:22.555470 25871 raft_consensus.cc:2229] T 67f7078192f043e3a6f238d133273386 P 1e0717d64bd54fffb6880899efbf0b0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:11:22.555792 25871 raft_consensus.cc:2258] T 67f7078192f043e3a6f238d133273386 P 1e0717d64bd54fffb6880899efbf0b0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:22.557351 25871 ts_tablet_manager.cc:1905] T 67f7078192f043e3a6f238d133273386 P 1e0717d64bd54fffb6880899efbf0b0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:22.557415 25873 ts_tablet_manager.cc:1918] T 67f7078192f043e3a6f238d133273386 P 07067b0671714947a205df92a8070061: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:22.557529 25873 log.cc:1196] T 67f7078192f043e3a6f238d133273386 P 07067b0671714947a205df92a8070061: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/67f7078192f043e3a6f238d133273386
I20260927 15:11:22.557864 25873 ts_tablet_manager.cc:1939] T 67f7078192f043e3a6f238d133273386 P 07067b0671714947a205df92a8070061: Deleting consensus metadata
I20260927 15:11:22.559324 25356 catalog_manager.cc:4841] TS 07067b0671714947a205df92a8070061 (127.7.17.195:38393): tablet 67f7078192f043e3a6f238d133273386 (table test-table [id=2c18a4b8bee542b8a9926ba7d7a99516]) successfully deleted
I20260927 15:11:22.560020 25871 ts_tablet_manager.cc:1918] T 67f7078192f043e3a6f238d133273386 P 1e0717d64bd54fffb6880899efbf0b0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:22.560155 25871 log.cc:1196] T 67f7078192f043e3a6f238d133273386 P 1e0717d64bd54fffb6880899efbf0b0b: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521636012660-7239-0/minicluster-data/ts-1/wal/wals/67f7078192f043e3a6f238d133273386
W20260927 15:11:22.560343 25356 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.17.193:35063 (error 108)
I20260927 15:11:22.560432 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 25566
I20260927 15:11:22.562538 25844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:22.562189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:35063 (local address 127.0.0.1:46774)
0927 15:11:22.562388 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:22.562454 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.193:35063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260927 15:11:22.562731 25356 catalog_manager.cc:4584] TS 9ccf6f8ab5764e459150792b5e5f14dd (127.7.17.193:35063): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67f7078192f043e3a6f238d133273386: Network error: Client connection negotiation failed: client connection to 127.7.17.193:35063: connect: Connection refused (error 111)
I20260927 15:11:22.566291 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 25700
I20260927 15:11:22.567437 25844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:22.566550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:36475 (local address 127.0.0.1:39274)
0927 15:11:22.567294 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:22.567353 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.194:36475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":654}
I20260927 15:11:22.571421 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 25338
2026-09-27T15:11:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2322 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T15:11:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:22Z Disabled control of system clock
I20260927 15:11:22.600963 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:43979
--webserver_interface=127.7.17.254
--webserver_port=0
--builtin_ntp_servers=127.7.17.212:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:43979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:11:22.713310 25878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:22.713605 25878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:22.713644 25878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:22.718186 25878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:22.718272 25878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:22.718287 25878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:22.718304 25878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:22.718319 25878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:22.723462 25878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:43979
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:43979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:22.724624 25878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:22.726181 25878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:22.734864 25884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:22.734895 25883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:22.735126 25878 server_base.cc:1034] running on GCE node
W20260927 15:11:22.734921 25886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:22.735785 25878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:22.737083 25878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:22.738255 25878 hybrid_clock.cc:648] HybridClock initialized: now 1790521882738235 us; error 32 us; skew 500 ppm
I20260927 15:11:22.740254 25878 webserver.cc:460] Webserver started at http://127.7.17.254:42195/ using document root <none> and password file <none>
I20260927 15:11:22.740876 25878 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:22.740944 25878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:22.741182 25878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:22.743424 25878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "728bd360a5364a58bb5181f8114a7d59"
format_stamp: "Formatted at 2026-09-27 15:11:22 on dist-test-slave-hhpq"
server_key: "36c9acea3e61552745f242909fe27327"
server_key_iv: "6bfac6809c56ca72f96b3d339cc8dfe9"
server_key_version: "encryptionkey@0"
I20260927 15:11:22.744045 25878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "728bd360a5364a58bb5181f8114a7d59"
format_stamp: "Formatted at 2026-09-27 15:11:22 on dist-test-slave-hhpq"
server_key: "36c9acea3e61552745f242909fe27327"
server_key_iv: "6bfac6809c56ca72f96b3d339cc8dfe9"
server_key_version: "encryptionkey@0"
I20260927 15:11:22.747826 25878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:11:22.750447 25892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:22.751775 25878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:11:22.751909 25878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "728bd360a5364a58bb5181f8114a7d59"
format_stamp: "Formatted at 2026-09-27 15:11:22 on dist-test-slave-hhpq"
server_key: "36c9acea3e61552745f242909fe27327"
server_key_iv: "6bfac6809c56ca72f96b3d339cc8dfe9"
server_key_version: "encryptionkey@0"
I20260927 15:11:22.752019 25878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:22.769028 25878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:22.769825 25878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:22.770025 25878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:22.777029 25878 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:43979
I20260927 15:11:22.777047 25944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:43979 every 8 connection(s)
I20260927 15:11:22.778044 25878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:22.781308 25945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:22.787356 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 25878
I20260927 15:11:22.787478 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:22.787467 25945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59: Bootstrap starting.
I20260927 15:11:22.787750 7239 external_mini_cluster.cc:1428] Setting key 1ce386c0144b7f0d6fd868bab5c8590d
I20260927 15:11:22.790334 25945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:22.791299 25945 log.cc:824] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:22.793895 25945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59: No bootstrap required, opened a new log
I20260927 15:11:22.798722 25945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "728bd360a5364a58bb5181f8114a7d59" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43979 } }
I20260927 15:11:22.799036 25945 raft_consensus.cc:374] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:22.799105 25945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 728bd360a5364a58bb5181f8114a7d59, State: Initialized, Role: FOLLOWER
I20260927 15:11:22.799494 25948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:22.789357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52972 (local address 127.7.17.254:43979)
0927 15:11:22.790082 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:22.790101 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:22.790148 (+ 47us) server_negotiation.cc:407] Connection header received
0927 15:11:22.791067 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:22.791095 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:22.791680 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:22.792059 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:22.792520 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.793965 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.794981 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:22.795725 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:22.795895 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:22.796785 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:22.796792 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:22.796811 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:22.797401 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:22.797483 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:22.797571 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:22.797667 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:22.797889 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 15:11:22.798228 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":113,"threads_started":1}
I20260927 15:11:22.799611 25945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "728bd360a5364a58bb5181f8114a7d59" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43979 } }
I20260927 15:11:22.799767 25945 raft_consensus.cc:388] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:22.799834 25945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:22.799932 25945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:22.801034 25945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "728bd360a5364a58bb5181f8114a7d59" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43979 } }
I20260927 15:11:22.801414 25945 leader_election.cc:302] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 728bd360a5364a58bb5181f8114a7d59; no voters:
I20260927 15:11:22.801774 25945 leader_election.cc:288] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:22.801880 25950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:22.802141 25950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59 [term 1 LEADER]: Becoming Leader. State: Replica: 728bd360a5364a58bb5181f8114a7d59, State: Running, Role: LEADER
I20260927 15:11:22.802492 25950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "728bd360a5364a58bb5181f8114a7d59" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43979 } }
I20260927 15:11:22.803035 25945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:22.804117 25952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 728bd360a5364a58bb5181f8114a7d59. Latest consensus state: current_term: 1 leader_uuid: "728bd360a5364a58bb5181f8114a7d59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "728bd360a5364a58bb5181f8114a7d59" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43979 } } }
I20260927 15:11:22.804245 25952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:22.804533 25951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "728bd360a5364a58bb5181f8114a7d59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "728bd360a5364a58bb5181f8114a7d59" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 43979 } } }
I20260927 15:11:22.804606 25959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:22.804703 25951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:22.807592 25959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:22.813369 25959 catalog_manager.cc:1347] Generated new cluster ID: c848cf65cb2f46a29dd3099664409712
I20260927 15:11:22.813452 25959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:22.826066 25959 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:22.827415 25959 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:22.833206 25959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59: Generated new TSK 0
I20260927 15:11:22.834102 25959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:22.841763 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43979
--builtin_ntp_servers=127.7.17.212:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:11:22.956494 25969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:22.956842 25969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:22.956890 25969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:22.961024 25969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:22.961087 25969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:22.961144 25969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:11:22.965960 25969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:22.967108 25969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:22.968478 25969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:22.978024 25975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:22.978068 25974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:22.978268 25969 server_base.cc:1034] running on GCE node
W20260927 15:11:22.978068 25977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:22.978772 25969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:22.979435 25969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:22.980641 25969 hybrid_clock.cc:648] HybridClock initialized: now 1790521882980622 us; error 33 us; skew 500 ppm
I20260927 15:11:22.982605 25969 webserver.cc:460] Webserver started at http://127.7.17.193:43435/ using document root <none> and password file <none>
I20260927 15:11:22.983176 25969 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:22.983242 25969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:22.983439 25969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:22.985608 25969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "f3fb7bb4102742d7b6b41e97a503534a"
format_stamp: "Formatted at 2026-09-27 15:11:22 on dist-test-slave-hhpq"
server_key: "093b1ba02994129f41d010cb257d08fa"
server_key_iv: "4eddd68f58a538e81840e0adf68e547c"
server_key_version: "encryptionkey@0"
I20260927 15:11:22.986173 25969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "f3fb7bb4102742d7b6b41e97a503534a"
format_stamp: "Formatted at 2026-09-27 15:11:22 on dist-test-slave-hhpq"
server_key: "093b1ba02994129f41d010cb257d08fa"
server_key_iv: "4eddd68f58a538e81840e0adf68e547c"
server_key_version: "encryptionkey@0"
I20260927 15:11:22.990164 25969 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:11:22.992841 25983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:22.994184 25969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:22.994305 25969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "f3fb7bb4102742d7b6b41e97a503534a"
format_stamp: "Formatted at 2026-09-27 15:11:22 on dist-test-slave-hhpq"
server_key: "093b1ba02994129f41d010cb257d08fa"
server_key_iv: "4eddd68f58a538e81840e0adf68e547c"
server_key_version: "encryptionkey@0"
I20260927 15:11:22.994414 25969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:23.016110 25969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:23.017006 25969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:23.017210 25969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:23.017853 25969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:23.018837 25969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:23.018887 25969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:23.018942 25969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:23.018972 25969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:23.027784 25969 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:36869
I20260927 15:11:23.027809 26096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:36869 every 8 connection(s)
I20260927 15:11:23.028867 25969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:11:23.038234 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 25969
I20260927 15:11:23.038388 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:11:23.038683 7239 external_mini_cluster.cc:1428] Setting key 2311318a03be38b56bfa3ae10f5722d0
I20260927 15:11:23.039521 25948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:23.030894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:52805 (local address 127.7.17.254:43979)
0927 15:11:23.031052 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:23.031057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:23.032347 (+ 1290us) server_negotiation.cc:407] Connection header received
0927 15:11:23.034204 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:23.034208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:23.034273 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:23.034350 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:23.035402 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.036219 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.037470 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.037715 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.037809 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:23.039017 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:23.039021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:23.039024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:23.039040 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:23.039065 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:23.039075 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:23.039144 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:23.039331 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 15:11:23.039399 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:11:23.040725 26099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:23.031370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:43979 (local address 127.7.17.193:52805)
0927 15:11:23.032161 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:23.032208 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:23.033856 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:23.034564 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:23.034572 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:23.034682 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:23.035256 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:23.035271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.036344 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.036347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:23.037347 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:23.037358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.037874 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.037878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:23.038058 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:23.038711 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:11:23.038713 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:23.038777 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:11:23.038866 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:23.039166 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:23.039176 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:23.039182 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:11:23.039315 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 15:11:23.039666 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":131,"threads_started":1}
I20260927 15:11:23.040776 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43979
--builtin_ntp_servers=127.7.17.212:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:11:23.043059 26097 heartbeater.cc:347] Connected to a master server at 127.7.17.254:43979
I20260927 15:11:23.043323 26097 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:23.043882 26097 heartbeater.cc:511] Master 127.7.17.254:43979 requested a full tablet report, sending...
I20260927 15:11:23.045480 25908 ts_manager.cc:194] Registered new tserver with Master: f3fb7bb4102742d7b6b41e97a503534a (127.7.17.193:36869)
I20260927 15:11:23.046657 25908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.17.193:52805
W20260927 15:11:23.154783 26100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:23.155129 26100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:23.155181 26100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:23.159516 26100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:23.159591 26100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:23.159648 26100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:11:23.164613 26100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:23.165789 26100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:23.167212 26100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:23.177138 26106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:23.177177 26100 server_base.cc:1034] running on GCE node
W20260927 15:11:23.177151 26108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:23.177294 26105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:23.177791 26100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:23.178529 26100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:23.179780 26100 hybrid_clock.cc:648] HybridClock initialized: now 1790521883179744 us; error 53 us; skew 500 ppm
I20260927 15:11:23.181957 26100 webserver.cc:460] Webserver started at http://127.7.17.194:44531/ using document root <none> and password file <none>
I20260927 15:11:23.182572 26100 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:23.182647 26100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:23.182863 26100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:23.185128 26100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "83044b24a72647ba9159671bc0a30646"
format_stamp: "Formatted at 2026-09-27 15:11:23 on dist-test-slave-hhpq"
server_key: "aacd07e7f29864e288680065f61347d3"
server_key_iv: "08eb94685fff4679448d5287fbc0b1eb"
server_key_version: "encryptionkey@0"
I20260927 15:11:23.185729 26100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "83044b24a72647ba9159671bc0a30646"
format_stamp: "Formatted at 2026-09-27 15:11:23 on dist-test-slave-hhpq"
server_key: "aacd07e7f29864e288680065f61347d3"
server_key_iv: "08eb94685fff4679448d5287fbc0b1eb"
server_key_version: "encryptionkey@0"
I20260927 15:11:23.189778 26100 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:11:23.192545 26114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:23.193954 26100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:11:23.194105 26100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "83044b24a72647ba9159671bc0a30646"
format_stamp: "Formatted at 2026-09-27 15:11:23 on dist-test-slave-hhpq"
server_key: "aacd07e7f29864e288680065f61347d3"
server_key_iv: "08eb94685fff4679448d5287fbc0b1eb"
server_key_version: "encryptionkey@0"
I20260927 15:11:23.194221 26100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:23.217181 26100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:23.217913 26100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:23.218108 26100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:23.218719 26100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:23.219693 26100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:23.219741 26100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:23.219797 26100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:23.219827 26100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:23.228588 26100 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:42829
I20260927 15:11:23.228744 26227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:42829 every 8 connection(s)
I20260927 15:11:23.229597 26100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:11:23.237614 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 26100
I20260927 15:11:23.237756 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:11:23.238095 7239 external_mini_cluster.cc:1428] Setting key 80e72dcdd8b24ec8a2422a4fdc396df9
I20260927 15:11:23.239440 25948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:23.231477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:45197 (local address 127.7.17.254:43979)
0927 15:11:23.231680 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:23.231685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:23.232888 (+ 1203us) server_negotiation.cc:407] Connection header received
0927 15:11:23.234332 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:23.234334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:23.234386 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:23.234466 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:23.235411 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.235959 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.237305 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.237541 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.237634 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:23.238941 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:23.238952 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:23.238955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:23.238970 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:23.238996 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:23.239007 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:23.239087 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:23.239239 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 15:11:23.239299 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 15:11:23.240271 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43979
--builtin_ntp_servers=127.7.17.212:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:11:23.240391 26230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:23.231963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:43979 (local address 127.7.17.194:45197)
0927 15:11:23.232711 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:23.232756 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:23.234105 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:23.234632 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:23.234639 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:23.234751 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:23.235266 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:23.235278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.236078 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.236081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:23.237168 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:23.237181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.237742 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.237747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:23.237965 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:23.238592 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:11:23.238593 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:23.238640 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:11:23.238732 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:23.239098 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:23.239106 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:23.239111 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:11:23.239227 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 15:11:23.239518 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":147,"threads_started":1}
I20260927 15:11:23.241670 26228 heartbeater.cc:347] Connected to a master server at 127.7.17.254:43979
I20260927 15:11:23.241973 26228 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:23.242626 26228 heartbeater.cc:511] Master 127.7.17.254:43979 requested a full tablet report, sending...
I20260927 15:11:23.243790 25908 ts_manager.cc:194] Registered new tserver with Master: 83044b24a72647ba9159671bc0a30646 (127.7.17.194:42829)
I20260927 15:11:23.244588 25908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.17.194:45197
W20260927 15:11:23.354198 26231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:23.354487 26231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:23.354525 26231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:23.358652 26231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:23.358716 26231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:23.358770 26231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:11:23.363581 26231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:43979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:23.364691 26231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:23.366247 26231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:23.376749 26237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:23.376796 26236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:23.376878 26231 server_base.cc:1034] running on GCE node
W20260927 15:11:23.376807 26239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:23.377489 26231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:23.378232 26231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:23.379420 26231 hybrid_clock.cc:648] HybridClock initialized: now 1790521883379395 us; error 41 us; skew 500 ppm
I20260927 15:11:23.381515 26231 webserver.cc:460] Webserver started at http://127.7.17.195:32833/ using document root <none> and password file <none>
I20260927 15:11:23.382185 26231 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:23.382261 26231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:23.382468 26231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:23.384634 26231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "b1e0f7aa330b4557acf53dad4bc1d476"
format_stamp: "Formatted at 2026-09-27 15:11:23 on dist-test-slave-hhpq"
server_key: "672625a48eaa249d9a5cc462dced9591"
server_key_iv: "1f2ea34402808ba2f53a7b2f3050b5d3"
server_key_version: "encryptionkey@0"
I20260927 15:11:23.385245 26231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "b1e0f7aa330b4557acf53dad4bc1d476"
format_stamp: "Formatted at 2026-09-27 15:11:23 on dist-test-slave-hhpq"
server_key: "672625a48eaa249d9a5cc462dced9591"
server_key_iv: "1f2ea34402808ba2f53a7b2f3050b5d3"
server_key_version: "encryptionkey@0"
I20260927 15:11:23.389189 26231 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:11:23.391868 26245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:23.393417 26231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:23.393548 26231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "b1e0f7aa330b4557acf53dad4bc1d476"
format_stamp: "Formatted at 2026-09-27 15:11:23 on dist-test-slave-hhpq"
server_key: "672625a48eaa249d9a5cc462dced9591"
server_key_iv: "1f2ea34402808ba2f53a7b2f3050b5d3"
server_key_version: "encryptionkey@0"
I20260927 15:11:23.393666 26231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:23.408587 26231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:23.409426 26231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:23.409624 26231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:23.410286 26231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:23.411301 26231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:23.411355 26231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:23.411412 26231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:23.411442 26231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:23.420812 26231 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:43861
I20260927 15:11:23.420866 26358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:43861 every 8 connection(s)
I20260927 15:11:23.421792 26231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:11:23.427433 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 26231
I20260927 15:11:23.427559 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:11:23.427863 7239 external_mini_cluster.cc:1428] Setting key 4d0c0f8ea4800eb7b076ee48f6c7bfbb
I20260927 15:11:23.432379 25948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:23.423764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:42023 (local address 127.7.17.254:43979)
0927 15:11:23.423899 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:23.423903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:23.425258 (+ 1355us) server_negotiation.cc:407] Connection header received
0927 15:11:23.426691 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:23.426694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:23.426751 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:23.426838 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:23.428290 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.428943 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.430415 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.430653 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.430739 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:23.431884 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:23.431887 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:23.431889 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:23.431903 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:23.431926 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:23.431934 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:23.431993 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:23.432178 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 15:11:23.432251 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:11:23.433311 26361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:23.424231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:43979 (local address 127.7.17.195:42023)
0927 15:11:23.425111 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:23.425158 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:23.426429 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:23.427102 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:23.427114 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:23.427245 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:23.428068 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:23.428097 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.429101 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.429105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:23.430282 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:23.430295 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.430786 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.430789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:23.430953 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:23.431587 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:11:23.431589 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:23.431636 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:11:23.431722 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:23.432003 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:23.432009 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:23.432015 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:11:23.432123 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 15:11:23.432392 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":130,"threads_started":1}
I20260927 15:11:23.434710 26359 heartbeater.cc:347] Connected to a master server at 127.7.17.254:43979
I20260927 15:11:23.435106 26359 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:23.435736 26359 heartbeater.cc:511] Master 127.7.17.254:43979 requested a full tablet report, sending...
I20260927 15:11:23.437247 25908 ts_manager.cc:194] Registered new tserver with Master: b1e0f7aa330b4557acf53dad4bc1d476 (127.7.17.195:43861)
I20260927 15:11:23.437845 25908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.17.195:42023
I20260927 15:11:23.442444 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:11:23.448609 25948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:23.444556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52974 (local address 127.7.17.254:43979)
0927 15:11:23.444689 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:23.444692 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:23.444780 (+ 88us) server_negotiation.cc:407] Connection header received
0927 15:11:23.444931 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:23.444934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:23.444979 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:23.445056 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:23.445351 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.445883 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.446731 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.446957 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.447040 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:23.448070 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:23.448074 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:23.448076 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:23.448091 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:23.448121 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:23.448130 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:23.448253 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:23.448438 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 15:11:23.448485 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:11:23.452023 25908 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:11:23.455265 25908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:11:23.470582 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:23.465051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51514 (local address 127.7.17.195:43861)
0927 15:11:23.465887 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:23.465896 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:23.465915 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:11:23.465978 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:23.465983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:23.466180 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:23.466353 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:23.466492 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.467748 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.469172 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.469775 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.469875 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:23.470130 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:23.470228 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 15:11:23.470375 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":102,"threads_started":1}
I20260927 15:11:23.470571 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:23.464509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:43861 (local address 127.0.0.1:51514)
0927 15:11:23.465262 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:23.465276 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:23.465362 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:23.466314 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:23.466316 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:23.466330 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:23.466397 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:23.466403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.467852 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.467856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:23.469006 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:23.469016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.469897 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.469901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:23.470017 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:23.470064 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 15:11:23.470198 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 15:11:23.470360 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":20,"thread_start_us":98,"threads_started":1}
I20260927 15:11:23.470728 26370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:23.464168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:42829 (local address 127.0.0.1:60046)
0927 15:11:23.464544 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:23.464776 (+ 232us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:23.465323 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:23.466043 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:23.466049 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:23.466073 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:23.466229 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:23.466236 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.467792 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.467795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:23.469059 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:23.469070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.470390 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.470393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:23.470471 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:23.470493 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:11:23.470564 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 15:11:23.470627 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":108,"thread_start_us":92,"threads_started":1}
I20260927 15:11:23.470573 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:23.464972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:36869 (local address 127.0.0.1:41964)
0927 15:11:23.465568 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:23.465581 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:23.465654 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:23.466194 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:23.466197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:23.466215 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:23.466317 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:23.466323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.467441 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.467445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:23.469095 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:23.469104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.470168 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.470171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:23.470255 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:23.470277 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:11:23.470324 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 15:11:23.470399 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":12,"thread_start_us":93,"threads_started":1}
I20260927 15:11:23.471170 26375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:23.465313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41964 (local address 127.7.17.193:36869)
0927 15:11:23.465760 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:23.465773 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:23.465797 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:11:23.465864 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:23.465869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:23.466046 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:23.466466 (+ 420us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:23.466599 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.467323 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.469215 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.470039 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.470637 (+ 598us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:23.470836 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:23.470890 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 15:11:23.470993 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":114,"threads_started":1}
I20260927 15:11:23.472216 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:23.464312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60046 (local address 127.7.17.194:42829)
0927 15:11:23.464784 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:23.464795 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:23.465039 (+ 244us) server_negotiation.cc:407] Connection header received
0927 15:11:23.465542 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:23.465548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:23.465878 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:23.466090 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:23.466362 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.467658 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.469205 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.470029 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.470137 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:23.470335 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:23.471048 (+ 713us) server_negotiation.cc:299] Negotiation successful
0927 15:11:23.471146 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":111,"threads_started":1}
I20260927 15:11:23.474429 26293 tablet_service.cc:1467] Processing CreateTablet for tablet 6ceda8bedab44f8081322b85933489d6 (DEFAULT_TABLE table=test-table [id=5913c59f4d5b4840b284addcfe4a852a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:23.475752 26293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ceda8bedab44f8081322b85933489d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:23.475409 26162 tablet_service.cc:1467] Processing CreateTablet for tablet 6ceda8bedab44f8081322b85933489d6 (DEFAULT_TABLE table=test-table [id=5913c59f4d5b4840b284addcfe4a852a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:23.476727 26162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ceda8bedab44f8081322b85933489d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:23.477381 26031 tablet_service.cc:1467] Processing CreateTablet for tablet 6ceda8bedab44f8081322b85933489d6 (DEFAULT_TABLE table=test-table [id=5913c59f4d5b4840b284addcfe4a852a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:23.478319 26031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ceda8bedab44f8081322b85933489d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:23.483767 26376 tablet_bootstrap.cc:492] T 6ceda8bedab44f8081322b85933489d6 P 83044b24a72647ba9159671bc0a30646: Bootstrap starting.
I20260927 15:11:23.484230 26377 tablet_bootstrap.cc:492] T 6ceda8bedab44f8081322b85933489d6 P b1e0f7aa330b4557acf53dad4bc1d476: Bootstrap starting.
I20260927 15:11:23.485217 26378 tablet_bootstrap.cc:492] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a: Bootstrap starting.
I20260927 15:11:23.486804 26377 tablet_bootstrap.cc:654] T 6ceda8bedab44f8081322b85933489d6 P b1e0f7aa330b4557acf53dad4bc1d476: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:23.486804 26376 tablet_bootstrap.cc:654] T 6ceda8bedab44f8081322b85933489d6 P 83044b24a72647ba9159671bc0a30646: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:23.486869 26378 tablet_bootstrap.cc:654] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:23.487701 26377 log.cc:824] T 6ceda8bedab44f8081322b85933489d6 P b1e0f7aa330b4557acf53dad4bc1d476: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:23.487701 26376 log.cc:824] T 6ceda8bedab44f8081322b85933489d6 P 83044b24a72647ba9159671bc0a30646: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:23.487715 26378 log.cc:824] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:23.489952 26376 tablet_bootstrap.cc:492] T 6ceda8bedab44f8081322b85933489d6 P 83044b24a72647ba9159671bc0a30646: No bootstrap required, opened a new log
I20260927 15:11:23.489965 26377 tablet_bootstrap.cc:492] T 6ceda8bedab44f8081322b85933489d6 P b1e0f7aa330b4557acf53dad4bc1d476: No bootstrap required, opened a new log
I20260927 15:11:23.490161 26377 ts_tablet_manager.cc:1397] T 6ceda8bedab44f8081322b85933489d6 P b1e0f7aa330b4557acf53dad4bc1d476: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:11:23.490172 26376 ts_tablet_manager.cc:1397] T 6ceda8bedab44f8081322b85933489d6 P 83044b24a72647ba9159671bc0a30646: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 15:11:23.490468 26378 tablet_bootstrap.cc:492] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a: No bootstrap required, opened a new log
I20260927 15:11:23.490669 26378 ts_tablet_manager.cc:1397] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 15:11:23.493682 26377 raft_consensus.cc:348] T 6ceda8bedab44f8081322b85933489d6 P b1e0f7aa330b4557acf53dad4bc1d476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83044b24a72647ba9159671bc0a30646" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42829 } } peers { permanent_uuid: "f3fb7bb4102742d7b6b41e97a503534a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 36869 } } peers { permanent_uuid: "b1e0f7aa330b4557acf53dad4bc1d476" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 43861 } }
I20260927 15:11:23.493942 26377 raft_consensus.cc:374] T 6ceda8bedab44f8081322b85933489d6 P b1e0f7aa330b4557acf53dad4bc1d476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:23.494014 26377 raft_consensus.cc:729] T 6ceda8bedab44f8081322b85933489d6 P b1e0f7aa330b4557acf53dad4bc1d476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e0f7aa330b4557acf53dad4bc1d476, State: Initialized, Role: FOLLOWER
I20260927 15:11:23.494563 26377 consensus_queue.cc:260] T 6ceda8bedab44f8081322b85933489d6 P b1e0f7aa330b4557acf53dad4bc1d476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83044b24a72647ba9159671bc0a30646" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42829 } } peers { permanent_uuid: "f3fb7bb4102742d7b6b41e97a503534a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 36869 } } peers { permanent_uuid: "b1e0f7aa330b4557acf53dad4bc1d476" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 43861 } }
I20260927 15:11:23.494448 26378 raft_consensus.cc:348] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83044b24a72647ba9159671bc0a30646" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42829 } } peers { permanent_uuid: "f3fb7bb4102742d7b6b41e97a503534a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 36869 } } peers { permanent_uuid: "b1e0f7aa330b4557acf53dad4bc1d476" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 43861 } }
I20260927 15:11:23.494583 26376 raft_consensus.cc:348] T 6ceda8bedab44f8081322b85933489d6 P 83044b24a72647ba9159671bc0a30646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83044b24a72647ba9159671bc0a30646" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42829 } } peers { permanent_uuid: "f3fb7bb4102742d7b6b41e97a503534a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 36869 } } peers { permanent_uuid: "b1e0f7aa330b4557acf53dad4bc1d476" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 43861 } }
I20260927 15:11:23.494736 26378 raft_consensus.cc:374] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:23.494791 26378 raft_consensus.cc:729] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3fb7bb4102742d7b6b41e97a503534a, State: Initialized, Role: FOLLOWER
I20260927 15:11:23.494835 26376 raft_consensus.cc:374] T 6ceda8bedab44f8081322b85933489d6 P 83044b24a72647ba9159671bc0a30646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:23.494905 26376 raft_consensus.cc:729] T 6ceda8bedab44f8081322b85933489d6 P 83044b24a72647ba9159671bc0a30646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83044b24a72647ba9159671bc0a30646, State: Initialized, Role: FOLLOWER
I20260927 15:11:23.495368 26378 consensus_queue.cc:260] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83044b24a72647ba9159671bc0a30646" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42829 } } peers { permanent_uuid: "f3fb7bb4102742d7b6b41e97a503534a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 36869 } } peers { permanent_uuid: "b1e0f7aa330b4557acf53dad4bc1d476" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 43861 } }
I20260927 15:11:23.495368 26376 consensus_queue.cc:260] T 6ceda8bedab44f8081322b85933489d6 P 83044b24a72647ba9159671bc0a30646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83044b24a72647ba9159671bc0a30646" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42829 } } peers { permanent_uuid: "f3fb7bb4102742d7b6b41e97a503534a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 36869 } } peers { permanent_uuid: "b1e0f7aa330b4557acf53dad4bc1d476" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 43861 } }
I20260927 15:11:23.495810 26377 ts_tablet_manager.cc:1428] T 6ceda8bedab44f8081322b85933489d6 P b1e0f7aa330b4557acf53dad4bc1d476: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 15:11:23.496089 26097 heartbeater.cc:503] Master 127.7.17.254:43979 was elected leader, sending a full tablet report...
I20260927 15:11:23.496148 26359 heartbeater.cc:503] Master 127.7.17.254:43979 was elected leader, sending a full tablet report...
I20260927 15:11:23.496397 26376 ts_tablet_manager.cc:1428] T 6ceda8bedab44f8081322b85933489d6 P 83044b24a72647ba9159671bc0a30646: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 15:11:23.496400 26378 ts_tablet_manager.cc:1428] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260927 15:11:23.497161 26228 heartbeater.cc:503] Master 127.7.17.254:43979 was elected leader, sending a full tablet report...
I20260927 15:11:23.505338 26383 raft_consensus.cc:482] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:11:23.505537 26383 raft_consensus.cc:504] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83044b24a72647ba9159671bc0a30646" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42829 } } peers { permanent_uuid: "f3fb7bb4102742d7b6b41e97a503534a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 36869 } } peers { permanent_uuid: "b1e0f7aa330b4557acf53dad4bc1d476" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 43861 } }
I20260927 15:11:23.506755 26383 leader_election.cc:288] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83044b24a72647ba9159671bc0a30646 (127.7.17.194:42829), b1e0f7aa330b4557acf53dad4bc1d476 (127.7.17.195:43861)
I20260927 15:11:23.510773 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:23.507402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:53837 (local address 127.7.17.194:42829)
0927 15:11:23.507533 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:23.507537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:23.507553 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:11:23.507608 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:23.507611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:23.507665 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:23.507742 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:23.508115 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.508632 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.509842 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.510249 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.510359 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:23.510406 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:23.510587 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 15:11:23.510653 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:11:23.510773 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:23.507196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:37473 (local address 127.7.17.195:43861)
0927 15:11:23.507375 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:23.507379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:23.507702 (+ 323us) server_negotiation.cc:407] Connection header received
0927 15:11:23.507791 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:23.507794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:23.507838 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:23.507934 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:23.508175 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.508674 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.509830 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.510224 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.510326 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:23.510380 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:23.510571 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 15:11:23.510641 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 15:11:23.510773 26099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:23.506988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:42829 (local address 127.7.17.193:53837)
0927 15:11:23.507120 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:23.507136 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:23.507231 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:23.507781 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:23.507785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:23.507806 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:23.507994 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:23.508004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.508737 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.508740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:23.509698 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:23.509715 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.510374 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.510377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:23.510473 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:23.510490 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 15:11:23.510559 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 15:11:23.510641 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":42}
I20260927 15:11:23.510785 26385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:23.507011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:43861 (local address 127.7.17.193:37473)
0927 15:11:23.507587 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:23.507599 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:23.507703 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:23.507939 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:23.507941 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:23.507955 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:23.508068 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:23.508075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.508778 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.508780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:23.509701 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:23.509712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.510341 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.510345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:23.510440 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:23.510464 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:11:23.510535 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 15:11:23.510618 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":241,"thread_start_us":129,"threads_started":1}
I20260927 15:11:23.511473 26182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ceda8bedab44f8081322b85933489d6" candidate_uuid: "f3fb7bb4102742d7b6b41e97a503534a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83044b24a72647ba9159671bc0a30646" is_pre_election: true
I20260927 15:11:23.511474 26313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ceda8bedab44f8081322b85933489d6" candidate_uuid: "f3fb7bb4102742d7b6b41e97a503534a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e0f7aa330b4557acf53dad4bc1d476" is_pre_election: true
I20260927 15:11:23.511772 26313 raft_consensus.cc:2454] T 6ceda8bedab44f8081322b85933489d6 P b1e0f7aa330b4557acf53dad4bc1d476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3fb7bb4102742d7b6b41e97a503534a in term 0.
I20260927 15:11:23.511773 26182 raft_consensus.cc:2454] T 6ceda8bedab44f8081322b85933489d6 P 83044b24a72647ba9159671bc0a30646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3fb7bb4102742d7b6b41e97a503534a in term 0.
I20260927 15:11:23.512248 25984 leader_election.cc:302] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1e0f7aa330b4557acf53dad4bc1d476, f3fb7bb4102742d7b6b41e97a503534a; no voters:
I20260927 15:11:23.512511 26383 raft_consensus.cc:2781] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:11:23.512594 26383 raft_consensus.cc:482] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:11:23.512642 26383 raft_consensus.cc:3037] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:23.513862 26383 raft_consensus.cc:504] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83044b24a72647ba9159671bc0a30646" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42829 } } peers { permanent_uuid: "f3fb7bb4102742d7b6b41e97a503534a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 36869 } } peers { permanent_uuid: "b1e0f7aa330b4557acf53dad4bc1d476" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 43861 } }
I20260927 15:11:23.514235 26383 leader_election.cc:288] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a [CANDIDATE]: Term 1 election: Requested vote from peers 83044b24a72647ba9159671bc0a30646 (127.7.17.194:42829), b1e0f7aa330b4557acf53dad4bc1d476 (127.7.17.195:43861)
I20260927 15:11:23.514693 26313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ceda8bedab44f8081322b85933489d6" candidate_uuid: "f3fb7bb4102742d7b6b41e97a503534a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e0f7aa330b4557acf53dad4bc1d476"
I20260927 15:11:23.514786 26182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ceda8bedab44f8081322b85933489d6" candidate_uuid: "f3fb7bb4102742d7b6b41e97a503534a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83044b24a72647ba9159671bc0a30646"
I20260927 15:11:23.514842 26313 raft_consensus.cc:3037] T 6ceda8bedab44f8081322b85933489d6 P b1e0f7aa330b4557acf53dad4bc1d476 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:23.514909 26182 raft_consensus.cc:3037] T 6ceda8bedab44f8081322b85933489d6 P 83044b24a72647ba9159671bc0a30646 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:23.516392 26313 raft_consensus.cc:2454] T 6ceda8bedab44f8081322b85933489d6 P b1e0f7aa330b4557acf53dad4bc1d476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3fb7bb4102742d7b6b41e97a503534a in term 1.
I20260927 15:11:23.516392 26182 raft_consensus.cc:2454] T 6ceda8bedab44f8081322b85933489d6 P 83044b24a72647ba9159671bc0a30646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3fb7bb4102742d7b6b41e97a503534a in term 1.
I20260927 15:11:23.516876 25984 leader_election.cc:302] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1e0f7aa330b4557acf53dad4bc1d476, f3fb7bb4102742d7b6b41e97a503534a; no voters:
I20260927 15:11:23.517093 26383 raft_consensus.cc:2781] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:23.517333 26383 raft_consensus.cc:686] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a [term 1 LEADER]: Becoming Leader. State: Replica: f3fb7bb4102742d7b6b41e97a503534a, State: Running, Role: LEADER
I20260927 15:11:23.517695 26383 consensus_queue.cc:237] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83044b24a72647ba9159671bc0a30646" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42829 } } peers { permanent_uuid: "f3fb7bb4102742d7b6b41e97a503534a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 36869 } } peers { permanent_uuid: "b1e0f7aa330b4557acf53dad4bc1d476" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 43861 } }
I20260927 15:11:23.521337 25909 catalog_manager.cc:5495] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a reported cstate change: term changed from 0 to 1, leader changed from <none> to f3fb7bb4102742d7b6b41e97a503534a (127.7.17.193). New cstate: current_term: 1 leader_uuid: "f3fb7bb4102742d7b6b41e97a503534a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83044b24a72647ba9159671bc0a30646" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3fb7bb4102742d7b6b41e97a503534a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 36869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1e0f7aa330b4557acf53dad4bc1d476" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 43861 } health_report { overall_health: UNKNOWN } } }
W20260927 15:11:23.530069 26098 tablet.cc:2369] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:11:23.536368 26375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:23.532620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41974 (local address 127.7.17.193:36869)
0927 15:11:23.532762 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:23.532775 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:23.532794 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:11:23.532900 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:23.532903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:23.532979 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:23.533160 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:11:23.533400 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.533906 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.534779 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:23.535029 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:23.535125 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:23.535484 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:11:23.535920 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:11:23.536067 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:23.536156 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 15:11:23.536195 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:11:23.542425 26313 raft_consensus.cc:1261] T 6ceda8bedab44f8081322b85933489d6 P b1e0f7aa330b4557acf53dad4bc1d476 [term 1 FOLLOWER]: Refusing update from remote peer f3fb7bb4102742d7b6b41e97a503534a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:23.542425 26182 raft_consensus.cc:1261] T 6ceda8bedab44f8081322b85933489d6 P 83044b24a72647ba9159671bc0a30646 [term 1 FOLLOWER]: Refusing update from remote peer f3fb7bb4102742d7b6b41e97a503534a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:23.543102 26383 consensus_queue.cc:1035] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e0f7aa330b4557acf53dad4bc1d476" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:23.543298 26386 consensus_queue.cc:1035] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a [LEADER]: Connected to new peer: Peer: permanent_uuid: "83044b24a72647ba9159671bc0a30646" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 42829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:23.549459 26389 mvcc.cc:205] Tried to move back new op lower bound from 7333977634982273024 to 7333977634891374592. Current Snapshot: MvccSnapshot[applied={T|T < 7333977634982273024}]
I20260927 15:11:23.550097 26390 mvcc.cc:205] Tried to move back new op lower bound from 7333977634982273024 to 7333977634891374592. Current Snapshot: MvccSnapshot[applied={T|T < 7333977634982273024}]
I20260927 15:11:23.553046 26391 mvcc.cc:205] Tried to move back new op lower bound from 7333977634982273024 to 7333977634891374592. Current Snapshot: MvccSnapshot[applied={T|T < 7333977634982273024}]
I20260927 15:11:23.560510 25907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:23.560714 25907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:23.563385 25907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 728bd360a5364a58bb5181f8114a7d59: Sending DeleteTablet for 3 replicas of tablet 6ceda8bedab44f8081322b85933489d6
I20260927 15:11:23.564352 26031 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ceda8bedab44f8081322b85933489d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:23 UTC) from {username='slave'} at 127.0.0.1:41964
I20260927 15:11:23.564352 26162 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ceda8bedab44f8081322b85933489d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:23 UTC) from {username='slave'} at 127.0.0.1:60046
I20260927 15:11:23.564380 26293 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ceda8bedab44f8081322b85933489d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:23 UTC) from {username='slave'} at 127.0.0.1:51514
I20260927 15:11:23.565083 26400 tablet_replica.cc:330] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a: stopping tablet replica
I20260927 15:11:23.565182 26399 tablet_replica.cc:330] T 6ceda8bedab44f8081322b85933489d6 P b1e0f7aa330b4557acf53dad4bc1d476: stopping tablet replica
I20260927 15:11:23.565517 26400 raft_consensus.cc:2229] T 6ceda8bedab44f8081322b85933489d6 P f3fb7bb4102742d7b6b41e97a503534a [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:11:23.565568 26399 raft_consensus.cc:2229] T 6ceda8bedab44f8081322b85933489d6 P b1e0f7aa330b4557acf53dad4bc1d476 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:11:23.565698 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 25969
I20260927 15:11:23.566612 26399 raft_consensus.cc:2258] T 6ceda8bedab44f8081322b85933489d6 P b1e0f7aa330b4557acf53dad4bc1d476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:23.567701 26401 tablet_replica.cc:330] T 6ceda8bedab44f8081322b85933489d6 P 83044b24a72647ba9159671bc0a30646: stopping tablet replica
I20260927 15:11:23.568089 26401 raft_consensus.cc:2229] T 6ceda8bedab44f8081322b85933489d6 P 83044b24a72647ba9159671bc0a30646 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:11:23.568320 26399 ts_tablet_manager.cc:1905] T 6ceda8bedab44f8081322b85933489d6 P b1e0f7aa330b4557acf53dad4bc1d476: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:23.568430 26401 raft_consensus.cc:2258] T 6ceda8bedab44f8081322b85933489d6 P 83044b24a72647ba9159671bc0a30646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:23.570076 26401 ts_tablet_manager.cc:1905] T 6ceda8bedab44f8081322b85933489d6 P 83044b24a72647ba9159671bc0a30646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:23.570722 26399 ts_tablet_manager.cc:1918] T 6ceda8bedab44f8081322b85933489d6 P b1e0f7aa330b4557acf53dad4bc1d476: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:23.570815 26399 log.cc:1196] T 6ceda8bedab44f8081322b85933489d6 P b1e0f7aa330b4557acf53dad4bc1d476: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/6ceda8bedab44f8081322b85933489d6
I20260927 15:11:23.571117 26399 ts_tablet_manager.cc:1939] T 6ceda8bedab44f8081322b85933489d6 P b1e0f7aa330b4557acf53dad4bc1d476: Deleting consensus metadata
I20260927 15:11:23.572424 25893 catalog_manager.cc:4841] TS b1e0f7aa330b4557acf53dad4bc1d476 (127.7.17.195:43861): tablet 6ceda8bedab44f8081322b85933489d6 (table test-table [id=5913c59f4d5b4840b284addcfe4a852a]) successfully deleted
I20260927 15:11:23.572638 26401 ts_tablet_manager.cc:1918] T 6ceda8bedab44f8081322b85933489d6 P 83044b24a72647ba9159671bc0a30646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:23.572733 26401 log.cc:1196] T 6ceda8bedab44f8081322b85933489d6 P 83044b24a72647ba9159671bc0a30646: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521636012660-7239-0/minicluster-data/ts-1/wal/wals/6ceda8bedab44f8081322b85933489d6
I20260927 15:11:23.573040 26401 ts_tablet_manager.cc:1939] T 6ceda8bedab44f8081322b85933489d6 P 83044b24a72647ba9159671bc0a30646: Deleting consensus metadata
W20260927 15:11:23.573525 25896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.17.193:36869 (error 108)
I20260927 15:11:23.574450 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 26100
I20260927 15:11:23.574589 25896 catalog_manager.cc:4841] TS 83044b24a72647ba9159671bc0a30646 (127.7.17.194:42829): tablet 6ceda8bedab44f8081322b85933489d6 (table test-table [id=5913c59f4d5b4840b284addcfe4a852a]) successfully deleted
I20260927 15:11:23.574718 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:23.574413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:36869 (local address 127.0.0.1:41982)
0927 15:11:23.574557 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:23.574634 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.193:36869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 15:11:23.574870 25896 catalog_manager.cc:4584] TS f3fb7bb4102742d7b6b41e97a503534a (127.7.17.193:36869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6ceda8bedab44f8081322b85933489d6: Network error: Client connection negotiation failed: client connection to 127.7.17.193:36869: connect: Connection refused (error 111)
I20260927 15:11:23.580097 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 26231
I20260927 15:11:23.585224 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 25878
2026-09-27T15:11:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1012 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:23 dist-test-slave-hhpq krb5kdc[26405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:23 dist-test-slave-hhpq krb5kdc[26405](info): set up 2 sockets
Sep 27 15:11:23 dist-test-slave-hhpq krb5kdc[26405](info): commencing operation
krb5kdc: starting...
W20260927 15:11:24.544675 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:11:24 dist-test-slave-hhpq krb5kdc[26405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:11:24.668613 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:42947
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:42947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:24.782660 26418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:24.782960 26418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:24.783004 26418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:24.787297 26418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:24.787382 26418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:24.787402 26418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:24.787417 26418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:24.787431 26418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:24.793164 26418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:42947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:42947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:24.794438 26418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:24.795519 26418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:24.804450 26423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:24.804445 26424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:24.804462 26426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:24.804816 26418 server_base.cc:1034] running on GCE node
I20260927 15:11:24.805474 26418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:24.806695 26418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:24.807868 26418 hybrid_clock.cc:648] HybridClock initialized: now 1790521884807836 us; error 44 us; skew 500 ppm
Sep 27 15:11:24 dist-test-slave-hhpq krb5kdc[26405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521884, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:24.810824 26418 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:11:24.811806 26418 webserver.cc:460] Webserver started at http://127.7.17.254:43161/ using document root <none> and password file <none>
I20260927 15:11:24.812372 26418 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:24.812443 26418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:24.812654 26418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:24.814868 26418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "6456ed39c17f457dbe8bf32f615da616"
format_stamp: "Formatted at 2026-09-27 15:11:24 on dist-test-slave-hhpq"
server_key: "d586c29ab7b8b7fe919c734e3d0d2c90"
server_key_iv: "9a88a7d8630bd4753bf43cb240c32a11"
server_key_version: "encryptionkey@0"
I20260927 15:11:24.815446 26418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "6456ed39c17f457dbe8bf32f615da616"
format_stamp: "Formatted at 2026-09-27 15:11:24 on dist-test-slave-hhpq"
server_key: "d586c29ab7b8b7fe919c734e3d0d2c90"
server_key_iv: "9a88a7d8630bd4753bf43cb240c32a11"
server_key_version: "encryptionkey@0"
I20260927 15:11:24.819376 26418 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:11:24.822103 26433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:24.823642 26418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:24.823801 26418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "6456ed39c17f457dbe8bf32f615da616"
format_stamp: "Formatted at 2026-09-27 15:11:24 on dist-test-slave-hhpq"
server_key: "d586c29ab7b8b7fe919c734e3d0d2c90"
server_key_iv: "9a88a7d8630bd4753bf43cb240c32a11"
server_key_version: "encryptionkey@0"
I20260927 15:11:24.823930 26418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:24.838014 26418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:24.841635 26418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:24.841827 26418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:24.848930 26418 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:42947
I20260927 15:11:24.848963 26485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:42947 every 8 connection(s)
I20260927 15:11:24.849938 26418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:24.852996 26486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:24.855258 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 26418
I20260927 15:11:24.855376 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:24.855643 7239 external_mini_cluster.cc:1428] Setting key fface8b09d929dd4bbb65964172706ba
I20260927 15:11:24.859295 26486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616: Bootstrap starting.
I20260927 15:11:24.862097 26486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:24.862985 26486 log.cc:824] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:24.865201 26486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616: No bootstrap required, opened a new log
Sep 27 15:11:24 dist-test-slave-hhpq krb5kdc[26405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:11:24.868785 26486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6456ed39c17f457dbe8bf32f615da616" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42947 } }
I20260927 15:11:24.869078 26486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:24.869141 26486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6456ed39c17f457dbe8bf32f615da616, State: Initialized, Role: FOLLOWER
I20260927 15:11:24.869655 26486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6456ed39c17f457dbe8bf32f615da616" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42947 } }
I20260927 15:11:24.869810 26486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:24.869880 26486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:24.869977 26486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:24.870534 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:24.857242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47106 (local address 127.7.17.254:42947)
0927 15:11:24.857689 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:24.857705 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:24.857743 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:11:24.858422 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:24.858443 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:24.858802 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:24.859114 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:24.860136 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:24.861250 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:24.862457 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:24.863060 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:24.863238 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:24.865852 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:24.865857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:24.865872 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:24.865901 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:24.867513 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:24.867941 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:24.867946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:24.867951 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:24.868127 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:24.868396 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:24.868399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:24.868400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:24.868872 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:24.869088 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:24.869361 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 15:11:24.869668 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":121,"threads_started":1}
I20260927 15:11:24.870980 26486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6456ed39c17f457dbe8bf32f615da616" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42947 } }
I20260927 15:11:24.871316 26486 leader_election.cc:302] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6456ed39c17f457dbe8bf32f615da616; no voters:
I20260927 15:11:24.871630 26486 leader_election.cc:288] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:24.871752 26491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:24.871965 26491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616 [term 1 LEADER]: Becoming Leader. State: Replica: 6456ed39c17f457dbe8bf32f615da616, State: Running, Role: LEADER
I20260927 15:11:24.872319 26491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6456ed39c17f457dbe8bf32f615da616" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42947 } }
I20260927 15:11:24.873005 26486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:24.873574 26493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6456ed39c17f457dbe8bf32f615da616. Latest consensus state: current_term: 1 leader_uuid: "6456ed39c17f457dbe8bf32f615da616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6456ed39c17f457dbe8bf32f615da616" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42947 } } }
I20260927 15:11:24.873703 26493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:24.874030 26492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6456ed39c17f457dbe8bf32f615da616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6456ed39c17f457dbe8bf32f615da616" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42947 } } }
I20260927 15:11:24.874184 26492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:24.874229 26497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:24.877297 26497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:24.883185 26497 catalog_manager.cc:1347] Generated new cluster ID: 42e2cf5ad2ee483094a2d8554d4dca16
I20260927 15:11:24.883270 26497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:24.899173 26497 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:24.900228 26497 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:24.908965 26497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616: Generated new TSK 0
I20260927 15:11:24.909791 26497 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
Entry for principal kudu/127.7.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab.
I20260927 15:11:24.945250 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:42947
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:25.060931 26512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:25.061220 26512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:25.061269 26512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:25.065436 26512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:25.065518 26512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:25.065588 26512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:11:25.071233 26512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.193.keytab
--principal=kudu/127.7.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:42947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:25.072381 26512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:25.073509 26512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:25.083573 26517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:25.083544 26520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:25.083547 26518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:25.083803 26512 server_base.cc:1034] running on GCE node
I20260927 15:11:25.084353 26512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:25.085037 26512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:25.086192 26512 hybrid_clock.cc:648] HybridClock initialized: now 1790521885086169 us; error 36 us; skew 500 ppm
Sep 27 15:11:25 dist-test-slave-hhpq krb5kdc[26405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521885, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:25.088963 26512 init.cc:377] Logged in from keytab as kudu/127.7.17.193@KRBTEST.COM (short username kudu)
I20260927 15:11:25.089902 26512 webserver.cc:460] Webserver started at http://127.7.17.193:38313/ using document root <none> and password file <none>
I20260927 15:11:25.090451 26512 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:25.090510 26512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:25.090713 26512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:25.092855 26512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "1bfa0446ca474455ad67e77a804b4292"
format_stamp: "Formatted at 2026-09-27 15:11:25 on dist-test-slave-hhpq"
server_key: "0785b7dea442946400f88f27aef625c3"
server_key_iv: "fff64b9b3107b0acf690f4bc6197495c"
server_key_version: "encryptionkey@0"
I20260927 15:11:25.093487 26512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "1bfa0446ca474455ad67e77a804b4292"
format_stamp: "Formatted at 2026-09-27 15:11:25 on dist-test-slave-hhpq"
server_key: "0785b7dea442946400f88f27aef625c3"
server_key_iv: "fff64b9b3107b0acf690f4bc6197495c"
server_key_version: "encryptionkey@0"
I20260927 15:11:25.097231 26512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:11:25.099751 26527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:25.101208 26512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:25.101325 26512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "1bfa0446ca474455ad67e77a804b4292"
format_stamp: "Formatted at 2026-09-27 15:11:25 on dist-test-slave-hhpq"
server_key: "0785b7dea442946400f88f27aef625c3"
server_key_iv: "fff64b9b3107b0acf690f4bc6197495c"
server_key_version: "encryptionkey@0"
I20260927 15:11:25.101438 26512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:25.113147 26512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:25.116365 26512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:25.116539 26512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:25.117187 26512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:25.118055 26512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:25.118104 26512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:25.118144 26512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:25.118161 26512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:25.127126 26512 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:37803
I20260927 15:11:25.127169 26640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:37803 every 8 connection(s)
I20260927 15:11:25.128212 26512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:11:25.131842 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 26512
I20260927 15:11:25.132007 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:11:25.132309 7239 external_mini_cluster.cc:1428] Setting key 2daf9df48e68be4e2ad2a50d84dc0fe9
Sep 27 15:11:25 dist-test-slave-hhpq krb5kdc[26405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521885, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.193@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.17.194@KRBTEST.COM; defaulting to no policy
I20260927 15:11:25.143702 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:25.130100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:43305 (local address 127.7.17.254:42947)
0927 15:11:25.130275 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:25.130279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:25.131264 (+ 985us) server_negotiation.cc:407] Connection header received
0927 15:11:25.132379 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:25.132383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:25.132444 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:25.132526 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:25.134070 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.134615 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.136097 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.136455 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.136569 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:25.139767 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:25.139770 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:25.139772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:25.139801 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:25.141287 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:25.141950 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:25.141952 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:25.141953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:25.142030 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:25.142402 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:25.142405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:25.142406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:25.142630 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:25.142750 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:25.143360 (+ 610us) server_negotiation.cc:299] Negotiation successful
0927 15:11:25.143501 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.194@KRBTEST.COM" created.
I20260927 15:11:25.144652 26643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:25.130565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:42947 (local address 127.7.17.193:43305)
0927 15:11:25.131097 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:25.131146 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:25.132132 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:25.132746 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:25.132755 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:25.133250 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:25.133919 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:25.133941 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.134748 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.134751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:25.135921 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:25.135933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.136700 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.136705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:25.136973 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:25.137634 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:25.137656 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:25.139502 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:25.141422 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:25.141429 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:25.141451 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:25.141852 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:25.142125 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:25.142129 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:25.142131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:25.142299 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:25.142752 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:25.142761 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:25.143079 (+ 318us) client_negotiation.cc:769] Sending connection context
0927 15:11:25.143306 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 15:11:25.143694 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":126,"threads_started":1}
I20260927 15:11:25.146027 26641 heartbeater.cc:347] Connected to a master server at 127.7.17.254:42947
I20260927 15:11:25.146301 26641 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:25.146842 26641 heartbeater.cc:511] Master 127.7.17.254:42947 requested a full tablet report, sending...
I20260927 15:11:25.148763 26450 ts_manager.cc:194] Registered new tserver with Master: 1bfa0446ca474455ad67e77a804b4292 (127.7.17.193:37803)
I20260927 15:11:25.150460 26450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.193@KRBTEST.COM'} at 127.7.17.193:43305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
Entry for principal kudu/127.7.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab.
I20260927 15:11:25.161908 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:42947
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:25.274735 26646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:25.275051 26646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:25.275116 26646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:25.279382 26646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:25.279461 26646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:25.279547 26646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:11:25.285045 26646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.194.keytab
--principal=kudu/127.7.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:42947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:25.286185 26646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:25.287245 26646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:25.297319 26652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:25.297319 26651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:25.297411 26654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:25.297595 26646 server_base.cc:1034] running on GCE node
I20260927 15:11:25.298063 26646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:25.298832 26646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:25.300041 26646 hybrid_clock.cc:648] HybridClock initialized: now 1790521885300005 us; error 59 us; skew 500 ppm
Sep 27 15:11:25 dist-test-slave-hhpq krb5kdc[26405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521885, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:25.303210 26646 init.cc:377] Logged in from keytab as kudu/127.7.17.194@KRBTEST.COM (short username kudu)
I20260927 15:11:25.304227 26646 webserver.cc:460] Webserver started at http://127.7.17.194:40957/ using document root <none> and password file <none>
I20260927 15:11:25.304888 26646 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:25.304961 26646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:25.305156 26646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:25.307400 26646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "1e4bb10499c3419bb215b71e6b440bf2"
format_stamp: "Formatted at 2026-09-27 15:11:25 on dist-test-slave-hhpq"
server_key: "4b6cbd4063a8b33b0689c9f0aa495bf4"
server_key_iv: "67dc76fdf71b0c8e864a13487e07105a"
server_key_version: "encryptionkey@0"
I20260927 15:11:25.307974 26646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "1e4bb10499c3419bb215b71e6b440bf2"
format_stamp: "Formatted at 2026-09-27 15:11:25 on dist-test-slave-hhpq"
server_key: "4b6cbd4063a8b33b0689c9f0aa495bf4"
server_key_iv: "67dc76fdf71b0c8e864a13487e07105a"
server_key_version: "encryptionkey@0"
I20260927 15:11:25.311945 26646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 15:11:25.314574 26661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:25.315949 26646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:11:25.316076 26646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "1e4bb10499c3419bb215b71e6b440bf2"
format_stamp: "Formatted at 2026-09-27 15:11:25 on dist-test-slave-hhpq"
server_key: "4b6cbd4063a8b33b0689c9f0aa495bf4"
server_key_iv: "67dc76fdf71b0c8e864a13487e07105a"
server_key_version: "encryptionkey@0"
I20260927 15:11:25.316195 26646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:25.326344 26646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:25.329752 26646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:25.329928 26646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:25.330518 26646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:25.331382 26646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:25.331435 26646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:25.331478 26646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:25.331501 26646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:25.340411 26646 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:40731
I20260927 15:11:25.340490 26774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:40731 every 8 connection(s)
I20260927 15:11:25.341441 26646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:11:25.348398 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 26646
I20260927 15:11:25.348519 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:11:25.348798 7239 external_mini_cluster.cc:1428] Setting key 6146976a498299112ca3e3da806371de
Sep 27 15:11:25 dist-test-slave-hhpq krb5kdc[26405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521885, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.194@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:11:25.356037 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:25.343518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:34891 (local address 127.7.17.254:42947)
0927 15:11:25.343667 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:25.343671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:25.344676 (+ 1005us) server_negotiation.cc:407] Connection header received
0927 15:11:25.345807 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:25.345810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:25.345860 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:25.345952 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:25.347280 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.347814 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.349135 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.349439 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.349549 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:25.352265 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:25.352269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:25.352271 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:25.352304 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:25.353732 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:25.354362 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:25.354365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:25.354366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:25.354442 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:25.354789 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:25.354792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:25.354794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:25.355058 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:25.355163 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:25.355717 (+ 554us) server_negotiation.cc:299] Negotiation successful
0927 15:11:25.355853 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:11:25.356890 26777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:25.343890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:42947 (local address 127.7.17.194:34891)
0927 15:11:25.344455 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:25.344532 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:25.345564 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:25.346181 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:25.346188 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:25.346582 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:25.347112 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:25.347127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.347937 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.347940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:25.349007 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:25.349018 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.349624 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.349628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:25.349810 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:25.350386 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:25.350405 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:25.352070 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:25.353855 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:25.353862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:25.353882 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:25.354264 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:25.354523 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:25.354526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:25.354528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:25.354694 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:25.355204 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:25.355210 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:25.355475 (+ 265us) client_negotiation.cc:769] Sending connection context
0927 15:11:25.355650 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 15:11:25.355934 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":126,"threads_started":1}
WARNING: no policy specified for kudu/127.7.17.195@KRBTEST.COM; defaulting to no policy
I20260927 15:11:25.358388 26775 heartbeater.cc:347] Connected to a master server at 127.7.17.254:42947
I20260927 15:11:25.358646 26775 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:25.359205 26775 heartbeater.cc:511] Master 127.7.17.254:42947 requested a full tablet report, sending...
I20260927 15:11:25.360308 26450 ts_manager.cc:194] Registered new tserver with Master: 1e4bb10499c3419bb215b71e6b440bf2 (127.7.17.194:40731)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.195@KRBTEST.COM" created.
I20260927 15:11:25.360963 26450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.194@KRBTEST.COM'} at 127.7.17.194:34891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
Entry for principal kudu/127.7.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab.
I20260927 15:11:25.377804 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:42947
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:25.488914 26780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:25.489209 26780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:25.489254 26780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:25.493455 26780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:25.493527 26780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:25.493608 26780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:11:25.499370 26780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.195.keytab
--principal=kudu/127.7.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:42947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:25.500504 26780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:25.501562 26780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:25.511628 26786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:25.511628 26785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:25.511667 26788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:25.511744 26780 server_base.cc:1034] running on GCE node
I20260927 15:11:25.512394 26780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:25.513113 26780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:25.514297 26780 hybrid_clock.cc:648] HybridClock initialized: now 1790521885514274 us; error 32 us; skew 500 ppm
Sep 27 15:11:25 dist-test-slave-hhpq krb5kdc[26405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521885, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:25.517220 26780 init.cc:377] Logged in from keytab as kudu/127.7.17.195@KRBTEST.COM (short username kudu)
I20260927 15:11:25.518190 26780 webserver.cc:460] Webserver started at http://127.7.17.195:37395/ using document root <none> and password file <none>
I20260927 15:11:25.518788 26780 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:25.518862 26780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:25.519064 26780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:25.521382 26780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "c2723f63005f438ea2cefd2e77ab7b81"
format_stamp: "Formatted at 2026-09-27 15:11:25 on dist-test-slave-hhpq"
server_key: "3fd6fa248105771d38526a57a3e03c3a"
server_key_iv: "3e0d9deaedd20f5be5415adc2acf3eaa"
server_key_version: "encryptionkey@0"
I20260927 15:11:25.521971 26780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "c2723f63005f438ea2cefd2e77ab7b81"
format_stamp: "Formatted at 2026-09-27 15:11:25 on dist-test-slave-hhpq"
server_key: "3fd6fa248105771d38526a57a3e03c3a"
server_key_iv: "3e0d9deaedd20f5be5415adc2acf3eaa"
server_key_version: "encryptionkey@0"
I20260927 15:11:25.526150 26780 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:11:25.529142 26795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:25.530560 26780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:11:25.530691 26780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "c2723f63005f438ea2cefd2e77ab7b81"
format_stamp: "Formatted at 2026-09-27 15:11:25 on dist-test-slave-hhpq"
server_key: "3fd6fa248105771d38526a57a3e03c3a"
server_key_iv: "3e0d9deaedd20f5be5415adc2acf3eaa"
server_key_version: "encryptionkey@0"
I20260927 15:11:25.530812 26780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:25.555353 26780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:25.558946 26780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:25.559163 26780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:25.559770 26780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:25.560619 26780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:25.560673 26780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:25.560729 26780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:25.560755 26780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:25.569403 26780 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:35089
I20260927 15:11:25.569445 26908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:35089 every 8 connection(s)
I20260927 15:11:25.570343 26780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:11:25.574618 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 26780
I20260927 15:11:25.574771 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:11:25.575095 7239 external_mini_cluster.cc:1428] Setting key 15fcd00eab2f5d371278407d89ca1610
Sep 27 15:11:25 dist-test-slave-hhpq krb5kdc[26405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521885, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.195@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:11:25.584583 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:25.572244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:54435 (local address 127.7.17.254:42947)
0927 15:11:25.572425 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:25.572430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:25.573397 (+ 967us) server_negotiation.cc:407] Connection header received
0927 15:11:25.574373 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:25.574376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:25.574428 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:25.574504 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:25.576129 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.576686 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.577940 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.578175 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.578268 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:25.580996 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:25.581000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:25.581003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:25.581036 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:25.582387 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:25.582946 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:25.582949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:25.582951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:25.583036 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:25.583392 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:25.583395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:25.583396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:25.583604 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:25.583691 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:25.584261 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 15:11:25.584399 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 15:11:25.585526 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:25.572660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:42947 (local address 127.7.17.195:54435)
0927 15:11:25.573239 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:25.573287 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:25.574150 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:25.574719 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:25.574730 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:25.575338 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:25.575981 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:25.576002 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.576818 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.576847 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:25.577812 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:25.577823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.578330 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.578334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:25.578534 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:25.579134 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:25.579152 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:25.580753 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:25.582518 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:25.582526 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:25.582547 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:25.582824 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:25.583121 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:25.583124 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:25.583126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:25.583290 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:25.583706 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:25.583715 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:25.583997 (+ 282us) client_negotiation.cc:769] Sending connection context
0927 15:11:25.584173 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 15:11:25.584553 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":152,"threads_started":1}
I20260927 15:11:25.586865 26909 heartbeater.cc:347] Connected to a master server at 127.7.17.254:42947
I20260927 15:11:25.587175 26909 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:25.587814 26909 heartbeater.cc:511] Master 127.7.17.254:42947 requested a full tablet report, sending...
I20260927 15:11:25.589076 26450 ts_manager.cc:194] Registered new tserver with Master: c2723f63005f438ea2cefd2e77ab7b81 (127.7.17.195:35089)
I20260927 15:11:25.589597 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:11:25.589836 26450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.17.195@KRBTEST.COM'} at 127.7.17.195:54435
I20260927 15:11:25.599193 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:25.591656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47112 (local address 127.7.17.254:42947)
0927 15:11:25.591785 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:25.591790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:25.591913 (+ 123us) server_negotiation.cc:407] Connection header received
0927 15:11:25.592008 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:25.592010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:25.592048 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:25.592120 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:25.592893 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.593420 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.594405 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.594640 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.594734 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:25.596213 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:25.596217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:25.596219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:25.596245 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:25.597557 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:25.597962 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:25.597965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:25.597966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:25.598080 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:25.598360 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:25.598363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:25.598364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:25.598594 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:25.598692 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:25.598880 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 15:11:25.598978 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:11:25.602460 26450 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:11:25.605720 26450 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:11:25.621136 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:25.615414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:37803 (local address 127.0.0.1:36292)
0927 15:11:25.616311 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:25.616325 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:25.616407 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:25.617177 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:25.617180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:25.617198 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:25.617272 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:25.617277 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.618323 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.618326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:25.619622 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:25.619632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.620462 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.620466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:25.620632 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:25.620680 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 15:11:25.620802 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 15:11:25.620974 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":94,"thread_start_us":82,"threads_started":1}
I20260927 15:11:25.621150 26925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:25.615816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36292 (local address 127.7.17.193:37803)
0927 15:11:25.616101 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:25.616116 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:25.616496 (+ 380us) server_negotiation.cc:407] Connection header received
0927 15:11:25.616575 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:25.616581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:25.616765 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:25.616951 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:25.617382 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.618193 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.619738 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.620340 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.620444 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:25.620733 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:25.620877 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 15:11:25.620984 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":82,"threads_started":1}
I20260927 15:11:25.621336 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:25.614666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:35089 (local address 127.0.0.1:50842)
0927 15:11:25.615477 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:25.615535 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:25.615691 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:25.616921 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:25.616924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:25.616942 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:25.617065 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:25.617072 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.618284 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.618287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:25.619666 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:25.619673 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.620933 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.620937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:25.621028 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:25.621051 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:11:25.621100 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 15:11:25.621192 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":188,"thread_start_us":82,"threads_started":1}
I20260927 15:11:25.621811 26923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:25.615175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45922 (local address 127.7.17.194:40731)
0927 15:11:25.615716 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:25.615729 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:25.615756 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:11:25.615839 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:25.615843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:25.616048 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:25.616215 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:25.616525 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.617565 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.619798 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.620766 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.620898 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:25.621040 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:25.621560 (+ 520us) server_negotiation.cc:299] Negotiation successful
0927 15:11:25.621660 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":120,"threads_started":1}
I20260927 15:11:25.621860 26921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:25.614772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50842 (local address 127.7.17.195:35089)
0927 15:11:25.616541 (+ 1769us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:25.616550 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:25.616568 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:11:25.616629 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:25.616632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:25.616780 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:25.616951 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:25.617180 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.618131 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.619798 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.620766 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.621443 (+ 677us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:25.621601 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:25.621646 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 15:11:25.621734 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1667,"thread_start_us":165,"threads_started":1}
I20260927 15:11:25.621470 26922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:25.615011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:40731 (local address 127.0.0.1:45922)
0927 15:11:25.615577 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:25.615592 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:25.615688 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:25.616217 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:25.616223 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:25.616245 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:25.616395 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:25.616401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.617704 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.617707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:25.619644 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:25.619656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.621099 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.621102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:25.621218 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:25.621239 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:11:25.621304 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 15:11:25.621369 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":44,"thread_start_us":83,"threads_started":1}
I20260927 15:11:25.624428 26575 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7c54a9392b426e84179067adc538e4 (DEFAULT_TABLE table=test-table [id=98cba2b4a3b74d08a59112ef0fa8f40b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:25.624630 26843 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7c54a9392b426e84179067adc538e4 (DEFAULT_TABLE table=test-table [id=98cba2b4a3b74d08a59112ef0fa8f40b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:25.624634 26709 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7c54a9392b426e84179067adc538e4 (DEFAULT_TABLE table=test-table [id=98cba2b4a3b74d08a59112ef0fa8f40b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:25.625501 26575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7c54a9392b426e84179067adc538e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:25.625710 26843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7c54a9392b426e84179067adc538e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:25.625710 26709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7c54a9392b426e84179067adc538e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:25.633517 26927 tablet_bootstrap.cc:492] T 8a7c54a9392b426e84179067adc538e4 P 1e4bb10499c3419bb215b71e6b440bf2: Bootstrap starting.
I20260927 15:11:25.636031 26927 tablet_bootstrap.cc:654] T 8a7c54a9392b426e84179067adc538e4 P 1e4bb10499c3419bb215b71e6b440bf2: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:25.636873 26927 log.cc:824] T 8a7c54a9392b426e84179067adc538e4 P 1e4bb10499c3419bb215b71e6b440bf2: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:25.637332 26926 tablet_bootstrap.cc:492] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292: Bootstrap starting.
I20260927 15:11:25.637614 26928 tablet_bootstrap.cc:492] T 8a7c54a9392b426e84179067adc538e4 P c2723f63005f438ea2cefd2e77ab7b81: Bootstrap starting.
I20260927 15:11:25.639344 26927 tablet_bootstrap.cc:492] T 8a7c54a9392b426e84179067adc538e4 P 1e4bb10499c3419bb215b71e6b440bf2: No bootstrap required, opened a new log
I20260927 15:11:25.639577 26927 ts_tablet_manager.cc:1397] T 8a7c54a9392b426e84179067adc538e4 P 1e4bb10499c3419bb215b71e6b440bf2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:11:25.639797 26928 tablet_bootstrap.cc:654] T 8a7c54a9392b426e84179067adc538e4 P c2723f63005f438ea2cefd2e77ab7b81: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:25.639846 26926 tablet_bootstrap.cc:654] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:25.640882 26928 log.cc:824] T 8a7c54a9392b426e84179067adc538e4 P c2723f63005f438ea2cefd2e77ab7b81: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:25.641109 26926 log.cc:824] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:25.642922 26926 tablet_bootstrap.cc:492] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292: No bootstrap required, opened a new log
I20260927 15:11:25.643069 26928 tablet_bootstrap.cc:492] T 8a7c54a9392b426e84179067adc538e4 P c2723f63005f438ea2cefd2e77ab7b81: No bootstrap required, opened a new log
I20260927 15:11:25.643132 26926 ts_tablet_manager.cc:1397] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:11:25.643267 26928 ts_tablet_manager.cc:1397] T 8a7c54a9392b426e84179067adc538e4 P c2723f63005f438ea2cefd2e77ab7b81: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:11:25.643666 26927 raft_consensus.cc:348] T 8a7c54a9392b426e84179067adc538e4 P 1e4bb10499c3419bb215b71e6b440bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2723f63005f438ea2cefd2e77ab7b81" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35089 } } peers { permanent_uuid: "1bfa0446ca474455ad67e77a804b4292" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37803 } } peers { permanent_uuid: "1e4bb10499c3419bb215b71e6b440bf2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 40731 } }
I20260927 15:11:25.643981 26927 raft_consensus.cc:374] T 8a7c54a9392b426e84179067adc538e4 P 1e4bb10499c3419bb215b71e6b440bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:25.644035 26927 raft_consensus.cc:729] T 8a7c54a9392b426e84179067adc538e4 P 1e4bb10499c3419bb215b71e6b440bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4bb10499c3419bb215b71e6b440bf2, State: Initialized, Role: FOLLOWER
I20260927 15:11:25.644579 26927 consensus_queue.cc:260] T 8a7c54a9392b426e84179067adc538e4 P 1e4bb10499c3419bb215b71e6b440bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2723f63005f438ea2cefd2e77ab7b81" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35089 } } peers { permanent_uuid: "1bfa0446ca474455ad67e77a804b4292" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37803 } } peers { permanent_uuid: "1e4bb10499c3419bb215b71e6b440bf2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 40731 } }
I20260927 15:11:25.645398 26775 heartbeater.cc:503] Master 127.7.17.254:42947 was elected leader, sending a full tablet report...
I20260927 15:11:25.645931 26927 ts_tablet_manager.cc:1428] T 8a7c54a9392b426e84179067adc538e4 P 1e4bb10499c3419bb215b71e6b440bf2: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260927 15:11:25.647022 26926 raft_consensus.cc:348] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2723f63005f438ea2cefd2e77ab7b81" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35089 } } peers { permanent_uuid: "1bfa0446ca474455ad67e77a804b4292" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37803 } } peers { permanent_uuid: "1e4bb10499c3419bb215b71e6b440bf2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 40731 } }
I20260927 15:11:25.647257 26926 raft_consensus.cc:374] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:25.647069 26928 raft_consensus.cc:348] T 8a7c54a9392b426e84179067adc538e4 P c2723f63005f438ea2cefd2e77ab7b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2723f63005f438ea2cefd2e77ab7b81" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35089 } } peers { permanent_uuid: "1bfa0446ca474455ad67e77a804b4292" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37803 } } peers { permanent_uuid: "1e4bb10499c3419bb215b71e6b440bf2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 40731 } }
I20260927 15:11:25.647318 26926 raft_consensus.cc:729] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bfa0446ca474455ad67e77a804b4292, State: Initialized, Role: FOLLOWER
I20260927 15:11:25.647326 26928 raft_consensus.cc:374] T 8a7c54a9392b426e84179067adc538e4 P c2723f63005f438ea2cefd2e77ab7b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:25.647373 26928 raft_consensus.cc:729] T 8a7c54a9392b426e84179067adc538e4 P c2723f63005f438ea2cefd2e77ab7b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2723f63005f438ea2cefd2e77ab7b81, State: Initialized, Role: FOLLOWER
I20260927 15:11:25.647773 26926 consensus_queue.cc:260] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2723f63005f438ea2cefd2e77ab7b81" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35089 } } peers { permanent_uuid: "1bfa0446ca474455ad67e77a804b4292" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37803 } } peers { permanent_uuid: "1e4bb10499c3419bb215b71e6b440bf2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 40731 } }
I20260927 15:11:25.647782 26928 consensus_queue.cc:260] T 8a7c54a9392b426e84179067adc538e4 P c2723f63005f438ea2cefd2e77ab7b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2723f63005f438ea2cefd2e77ab7b81" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35089 } } peers { permanent_uuid: "1bfa0446ca474455ad67e77a804b4292" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37803 } } peers { permanent_uuid: "1e4bb10499c3419bb215b71e6b440bf2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 40731 } }
I20260927 15:11:25.648367 26641 heartbeater.cc:503] Master 127.7.17.254:42947 was elected leader, sending a full tablet report...
I20260927 15:11:25.648753 26926 ts_tablet_manager.cc:1428] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:11:25.648754 26928 ts_tablet_manager.cc:1428] T 8a7c54a9392b426e84179067adc538e4 P c2723f63005f438ea2cefd2e77ab7b81: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 15:11:25.649109 26909 heartbeater.cc:503] Master 127.7.17.254:42947 was elected leader, sending a full tablet report...
I20260927 15:11:25.666086 26933 raft_consensus.cc:482] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:11:25.666296 26933 raft_consensus.cc:504] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2723f63005f438ea2cefd2e77ab7b81" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35089 } } peers { permanent_uuid: "1bfa0446ca474455ad67e77a804b4292" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37803 } } peers { permanent_uuid: "1e4bb10499c3419bb215b71e6b440bf2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 40731 } }
I20260927 15:11:25.667559 26933 leader_election.cc:288] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2723f63005f438ea2cefd2e77ab7b81 (127.7.17.195:35089), 1e4bb10499c3419bb215b71e6b440bf2 (127.7.17.194:40731)
I20260927 15:11:25.672263 26921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:25.667937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:53715 (local address 127.7.17.195:35089)
0927 15:11:25.668069 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:25.668074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:25.668413 (+ 339us) server_negotiation.cc:407] Connection header received
0927 15:11:25.668520 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:25.668524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:25.668592 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:25.668674 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:25.669066 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.669864 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.671264 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.671670 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.671777 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:25.671831 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:25.672080 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 15:11:25.672153 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:11:25.672258 26923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:25.667938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:44769 (local address 127.7.17.194:40731)
0927 15:11:25.668075 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:25.668079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:25.668367 (+ 288us) server_negotiation.cc:407] Connection header received
0927 15:11:25.668533 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:25.668535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:25.668597 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:25.668675 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:25.669077 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.669877 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.671249 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.671667 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.671777 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:25.671830 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:25.672062 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 15:11:25.672136 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:11:25.672276 26935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:25.667834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:35089 (local address 127.7.17.193:53715)
0927 15:11:25.668329 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:25.668340 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:25.668413 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:25.668702 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:25.668704 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:25.668716 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:25.668945 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:25.668953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.670006 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.670010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:25.671128 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:25.671141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.671786 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.671789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:25.671941 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:25.671977 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:11:25.672048 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 15:11:25.672119 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":160,"spinlock_wait_cycles":3840,"thread_start_us":90,"threads_started":1}
I20260927 15:11:25.672281 26936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:25.667834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:40731 (local address 127.7.17.193:44769)
0927 15:11:25.668260 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:25.668277 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:25.668409 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:25.668692 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:25.668695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:25.668710 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:25.668945 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:25.668953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.670024 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.670028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:25.671128 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:25.671140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.671785 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.671788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:25.671930 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:25.671959 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 15:11:25.672016 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 15:11:25.672103 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":51,"thread_start_us":119,"threads_started":1}
I20260927 15:11:25.672976 26863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7c54a9392b426e84179067adc538e4" candidate_uuid: "1bfa0446ca474455ad67e77a804b4292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2723f63005f438ea2cefd2e77ab7b81" is_pre_election: true
I20260927 15:11:25.673393 26863 raft_consensus.cc:2454] T 8a7c54a9392b426e84179067adc538e4 P c2723f63005f438ea2cefd2e77ab7b81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bfa0446ca474455ad67e77a804b4292 in term 0.
I20260927 15:11:25.673442 26729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7c54a9392b426e84179067adc538e4" candidate_uuid: "1bfa0446ca474455ad67e77a804b4292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4bb10499c3419bb215b71e6b440bf2" is_pre_election: true
I20260927 15:11:25.673733 26729 raft_consensus.cc:2454] T 8a7c54a9392b426e84179067adc538e4 P 1e4bb10499c3419bb215b71e6b440bf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bfa0446ca474455ad67e77a804b4292 in term 0.
I20260927 15:11:25.673894 26531 leader_election.cc:302] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bfa0446ca474455ad67e77a804b4292, c2723f63005f438ea2cefd2e77ab7b81; no voters:
I20260927 15:11:25.674191 26933 raft_consensus.cc:2781] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:11:25.674273 26933 raft_consensus.cc:482] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:11:25.674324 26933 raft_consensus.cc:3037] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:25.675443 26933 raft_consensus.cc:504] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2723f63005f438ea2cefd2e77ab7b81" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35089 } } peers { permanent_uuid: "1bfa0446ca474455ad67e77a804b4292" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37803 } } peers { permanent_uuid: "1e4bb10499c3419bb215b71e6b440bf2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 40731 } }
I20260927 15:11:25.675796 26933 leader_election.cc:288] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292 [CANDIDATE]: Term 1 election: Requested vote from peers c2723f63005f438ea2cefd2e77ab7b81 (127.7.17.195:35089), 1e4bb10499c3419bb215b71e6b440bf2 (127.7.17.194:40731)
I20260927 15:11:25.676185 26863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7c54a9392b426e84179067adc538e4" candidate_uuid: "1bfa0446ca474455ad67e77a804b4292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2723f63005f438ea2cefd2e77ab7b81"
I20260927 15:11:25.676317 26863 raft_consensus.cc:3037] T 8a7c54a9392b426e84179067adc538e4 P c2723f63005f438ea2cefd2e77ab7b81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:25.676333 26729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7c54a9392b426e84179067adc538e4" candidate_uuid: "1bfa0446ca474455ad67e77a804b4292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4bb10499c3419bb215b71e6b440bf2"
I20260927 15:11:25.676450 26729 raft_consensus.cc:3037] T 8a7c54a9392b426e84179067adc538e4 P 1e4bb10499c3419bb215b71e6b440bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:25.677675 26863 raft_consensus.cc:2454] T 8a7c54a9392b426e84179067adc538e4 P c2723f63005f438ea2cefd2e77ab7b81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bfa0446ca474455ad67e77a804b4292 in term 1.
I20260927 15:11:25.677698 26729 raft_consensus.cc:2454] T 8a7c54a9392b426e84179067adc538e4 P 1e4bb10499c3419bb215b71e6b440bf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bfa0446ca474455ad67e77a804b4292 in term 1.
I20260927 15:11:25.677980 26531 leader_election.cc:302] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bfa0446ca474455ad67e77a804b4292, c2723f63005f438ea2cefd2e77ab7b81; no voters:
I20260927 15:11:25.678174 26933 raft_consensus.cc:2781] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:25.678392 26933 raft_consensus.cc:686] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292 [term 1 LEADER]: Becoming Leader. State: Replica: 1bfa0446ca474455ad67e77a804b4292, State: Running, Role: LEADER
I20260927 15:11:25.678771 26933 consensus_queue.cc:237] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2723f63005f438ea2cefd2e77ab7b81" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35089 } } peers { permanent_uuid: "1bfa0446ca474455ad67e77a804b4292" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37803 } } peers { permanent_uuid: "1e4bb10499c3419bb215b71e6b440bf2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 40731 } }
I20260927 15:11:25.682194 26449 catalog_manager.cc:5495] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bfa0446ca474455ad67e77a804b4292 (127.7.17.193). New cstate: current_term: 1 leader_uuid: "1bfa0446ca474455ad67e77a804b4292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2723f63005f438ea2cefd2e77ab7b81" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bfa0446ca474455ad67e77a804b4292" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e4bb10499c3419bb215b71e6b440bf2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 40731 } health_report { overall_health: UNKNOWN } } }
I20260927 15:11:25.708252 26925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:25.704697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36308 (local address 127.7.17.193:37803)
0927 15:11:25.704858 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:25.704863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:25.704878 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:11:25.705004 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:25.705007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:25.705070 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:25.705168 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:11:25.705418 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.705953 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.706808 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.707040 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.707126 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:25.707440 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:11:25.707867 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:11:25.707972 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:25.708045 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 15:11:25.708107 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 15:11:25.714258 26863 raft_consensus.cc:1261] T 8a7c54a9392b426e84179067adc538e4 P c2723f63005f438ea2cefd2e77ab7b81 [term 1 FOLLOWER]: Refusing update from remote peer 1bfa0446ca474455ad67e77a804b4292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:25.714257 26729 raft_consensus.cc:1261] T 8a7c54a9392b426e84179067adc538e4 P 1e4bb10499c3419bb215b71e6b440bf2 [term 1 FOLLOWER]: Refusing update from remote peer 1bfa0446ca474455ad67e77a804b4292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:25.715005 26937 consensus_queue.cc:1035] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e4bb10499c3419bb215b71e6b440bf2" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 40731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:25.715220 26933 consensus_queue.cc:1035] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2723f63005f438ea2cefd2e77ab7b81" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:25.722781 26940 mvcc.cc:205] Tried to move back new op lower bound from 7333977643878297600 to 7333977643742978048. Current Snapshot: MvccSnapshot[applied={T|T < 7333977643878297600}]
I20260927 15:11:25.724040 26942 mvcc.cc:205] Tried to move back new op lower bound from 7333977643878297600 to 7333977643742978048. Current Snapshot: MvccSnapshot[applied={T|T < 7333977643878297600}]
I20260927 15:11:25.725878 26941 mvcc.cc:205] Tried to move back new op lower bound from 7333977643878297600 to 7333977643742978048. Current Snapshot: MvccSnapshot[applied={T|T < 7333977643878297600}]
I20260927 15:11:25.732700 26449 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:25.732937 26449 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:25.735518 26449 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6456ed39c17f457dbe8bf32f615da616: Sending DeleteTablet for 3 replicas of tablet 8a7c54a9392b426e84179067adc538e4
I20260927 15:11:25.736425 26709 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a7c54a9392b426e84179067adc538e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:25 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:45922
I20260927 15:11:25.736429 26575 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a7c54a9392b426e84179067adc538e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:25 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:36292
I20260927 15:11:25.736613 26843 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a7c54a9392b426e84179067adc538e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:25 UTC) from {username='kudu', principal='kudu/127.7.17.254@KRBTEST.COM'} at 127.0.0.1:50842
I20260927 15:11:25.736984 26949 tablet_replica.cc:330] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292: stopping tablet replica
I20260927 15:11:25.736984 26950 tablet_replica.cc:330] T 8a7c54a9392b426e84179067adc538e4 P 1e4bb10499c3419bb215b71e6b440bf2: stopping tablet replica
I20260927 15:11:25.737311 26950 raft_consensus.cc:2229] T 8a7c54a9392b426e84179067adc538e4 P 1e4bb10499c3419bb215b71e6b440bf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:11:25.737311 26949 raft_consensus.cc:2229] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:11:25.737638 26950 raft_consensus.cc:2258] T 8a7c54a9392b426e84179067adc538e4 P 1e4bb10499c3419bb215b71e6b440bf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:25.737712 26949 raft_consensus.cc:2258] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:25.739151 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 26512
I20260927 15:11:25.739162 26949 ts_tablet_manager.cc:1905] T 8a7c54a9392b426e84179067adc538e4 P 1bfa0446ca474455ad67e77a804b4292: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:25.740068 26950 ts_tablet_manager.cc:1905] T 8a7c54a9392b426e84179067adc538e4 P 1e4bb10499c3419bb215b71e6b440bf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:25.742368 26950 ts_tablet_manager.cc:1918] T 8a7c54a9392b426e84179067adc538e4 P 1e4bb10499c3419bb215b71e6b440bf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:25.742465 26950 log.cc:1196] T 8a7c54a9392b426e84179067adc538e4 P 1e4bb10499c3419bb215b71e6b440bf2: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-1/wal/wals/8a7c54a9392b426e84179067adc538e4
I20260927 15:11:25.742719 26950 ts_tablet_manager.cc:1939] T 8a7c54a9392b426e84179067adc538e4 P 1e4bb10499c3419bb215b71e6b440bf2: Deleting consensus metadata
I20260927 15:11:25.743090 26951 tablet_replica.cc:330] T 8a7c54a9392b426e84179067adc538e4 P c2723f63005f438ea2cefd2e77ab7b81: stopping tablet replica
I20260927 15:11:25.743440 26951 raft_consensus.cc:2229] T 8a7c54a9392b426e84179067adc538e4 P c2723f63005f438ea2cefd2e77ab7b81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:11:25.743786 26951 raft_consensus.cc:2258] T 8a7c54a9392b426e84179067adc538e4 P c2723f63005f438ea2cefd2e77ab7b81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:25.744194 26435 catalog_manager.cc:4841] TS 1e4bb10499c3419bb215b71e6b440bf2 (127.7.17.194:40731): tablet 8a7c54a9392b426e84179067adc538e4 (table test-table [id=98cba2b4a3b74d08a59112ef0fa8f40b]) successfully deleted
I20260927 15:11:25.745345 26951 ts_tablet_manager.cc:1905] T 8a7c54a9392b426e84179067adc538e4 P c2723f63005f438ea2cefd2e77ab7b81: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 15:11:25.745723 26437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.17.193:37803 (error 108)
I20260927 15:11:25.746307 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 26646
I20260927 15:11:25.747434 26922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:25.746500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:37803 (local address 127.0.0.1:36322)
0927 15:11:25.747237 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:25.747339 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.193:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":639}
I20260927 15:11:25.748106 26951 ts_tablet_manager.cc:1918] T 8a7c54a9392b426e84179067adc538e4 P c2723f63005f438ea2cefd2e77ab7b81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:25.748209 26951 log.cc:1196] T 8a7c54a9392b426e84179067adc538e4 P c2723f63005f438ea2cefd2e77ab7b81: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/8a7c54a9392b426e84179067adc538e4
W20260927 15:11:25.748402 26437 catalog_manager.cc:4584] TS 1bfa0446ca474455ad67e77a804b4292 (127.7.17.193:37803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a7c54a9392b426e84179067adc538e4: Network error: Client connection negotiation failed: client connection to 127.7.17.193:37803: connect: Connection refused (error 111)
I20260927 15:11:25.748538 26951 ts_tablet_manager.cc:1939] T 8a7c54a9392b426e84179067adc538e4 P c2723f63005f438ea2cefd2e77ab7b81: Deleting consensus metadata
I20260927 15:11:25.749780 26437 catalog_manager.cc:4841] TS c2723f63005f438ea2cefd2e77ab7b81 (127.7.17.195:35089): tablet 8a7c54a9392b426e84179067adc538e4 (table test-table [id=98cba2b4a3b74d08a59112ef0fa8f40b]) successfully deleted
I20260927 15:11:25.751967 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 26780
I20260927 15:11:25.757172 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 26418
2026-09-27T15:11:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2172 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T15:11:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:25Z Disabled control of system clock
I20260927 15:11:25.785459 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:46527
--webserver_interface=127.7.17.254
--webserver_port=0
--builtin_ntp_servers=127.7.17.212:44121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:46527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:11:25.897269 26956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:25.897590 26956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:25.897629 26956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:25.901887 26956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:25.901969 26956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:25.901984 26956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:25.901999 26956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:25.902014 26956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:25.907124 26956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:44121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:46527
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:46527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:25.908262 26956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:25.909716 26956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:25.918848 26962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:25.918848 26961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:25.918851 26964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:25.919016 26956 server_base.cc:1034] running on GCE node
I20260927 15:11:25.919845 26956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:25.921176 26956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:25.922384 26956 hybrid_clock.cc:648] HybridClock initialized: now 1790521885922356 us; error 43 us; skew 500 ppm
I20260927 15:11:25.924732 26956 webserver.cc:460] Webserver started at http://127.7.17.254:46797/ using document root <none> and password file <none>
I20260927 15:11:25.925413 26956 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:25.925500 26956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:25.925685 26956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:25.927937 26956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "073e668a7d7348389b4942eb5370366f"
format_stamp: "Formatted at 2026-09-27 15:11:25 on dist-test-slave-hhpq"
server_key: "f251af1558702c39f7f4d5318672f737"
server_key_iv: "58c00a264e1dbe4b127302b1d86cd0ec"
server_key_version: "encryptionkey@0"
I20260927 15:11:25.928544 26956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "073e668a7d7348389b4942eb5370366f"
format_stamp: "Formatted at 2026-09-27 15:11:25 on dist-test-slave-hhpq"
server_key: "f251af1558702c39f7f4d5318672f737"
server_key_iv: "58c00a264e1dbe4b127302b1d86cd0ec"
server_key_version: "encryptionkey@0"
I20260927 15:11:25.932687 26956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 15:11:25.935678 26970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:25.937387 26956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:25.937542 26956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "073e668a7d7348389b4942eb5370366f"
format_stamp: "Formatted at 2026-09-27 15:11:25 on dist-test-slave-hhpq"
server_key: "f251af1558702c39f7f4d5318672f737"
server_key_iv: "58c00a264e1dbe4b127302b1d86cd0ec"
server_key_version: "encryptionkey@0"
I20260927 15:11:25.937695 26956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:25.947794 26956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:25.948532 26956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:25.948714 26956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:25.955513 26956 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:46527
I20260927 15:11:25.955533 27022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:46527 every 8 connection(s)
I20260927 15:11:25.956525 26956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:25.959939 27023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:25.961292 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 26956
I20260927 15:11:25.961411 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:25.961694 7239 external_mini_cluster.cc:1428] Setting key d87b853f725a0613dddeff1bac58dd1d
I20260927 15:11:25.966295 27023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f: Bootstrap starting.
I20260927 15:11:25.969126 27023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:25.970000 27023 log.cc:824] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:25.971697 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:25.963144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43812 (local address 127.7.17.254:46527)
0927 15:11:25.963804 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:25.963821 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:25.963860 (+ 39us) server_negotiation.cc:407] Connection header received
0927 15:11:25.964521 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:25.964540 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:25.965051 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:25.965357 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:25.965725 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.966754 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.968012 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:25.968533 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:25.968683 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:25.969671 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:25.969677 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:25.969693 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:25.970123 (+ 430us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:25.970207 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:25.970232 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:25.970317 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:25.970558 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 15:11:25.970868 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":118,"threads_started":1}
I20260927 15:11:25.972379 27023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f: No bootstrap required, opened a new log
I20260927 15:11:25.976322 27023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073e668a7d7348389b4942eb5370366f" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46527 } }
I20260927 15:11:25.976588 27023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:25.976645 27023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 073e668a7d7348389b4942eb5370366f, State: Initialized, Role: FOLLOWER
I20260927 15:11:25.977129 27023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073e668a7d7348389b4942eb5370366f" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46527 } }
I20260927 15:11:25.977269 27023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:25.977339 27023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:25.977440 27023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:25.978652 27023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073e668a7d7348389b4942eb5370366f" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46527 } }
I20260927 15:11:25.978977 27023 leader_election.cc:302] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 073e668a7d7348389b4942eb5370366f; no voters:
I20260927 15:11:25.979281 27023 leader_election.cc:288] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:25.979410 27028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:25.979637 27028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f [term 1 LEADER]: Becoming Leader. State: Replica: 073e668a7d7348389b4942eb5370366f, State: Running, Role: LEADER
I20260927 15:11:25.979976 27028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073e668a7d7348389b4942eb5370366f" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46527 } }
I20260927 15:11:25.980536 27023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:25.981753 27029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "073e668a7d7348389b4942eb5370366f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073e668a7d7348389b4942eb5370366f" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46527 } } }
I20260927 15:11:25.981912 27029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:25.981772 27030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 073e668a7d7348389b4942eb5370366f. Latest consensus state: current_term: 1 leader_uuid: "073e668a7d7348389b4942eb5370366f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073e668a7d7348389b4942eb5370366f" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 46527 } } }
I20260927 15:11:25.982142 27030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:25.982321 27037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:25.985541 27037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:25.991230 27037 catalog_manager.cc:1347] Generated new cluster ID: d38292ba92a944eda4152baaff0bda4c
I20260927 15:11:25.991323 27037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:26.002748 27037 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:26.004073 27037 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:26.014006 27037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f: Generated new TSK 0
I20260927 15:11:26.015231 27037 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:26.027325 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.193:0
--local_ip_for_outbound_sockets=127.7.17.193
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46527
--builtin_ntp_servers=127.7.17.212:44121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:11:26.141480 27047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:26.141788 27047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:26.141842 27047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:26.146152 27047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:26.146231 27047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:26.146315 27047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.193
I20260927 15:11:26.151377 27047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:44121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.17.193
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.193
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:26.152540 27047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:26.154258 27047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:26.164209 27052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:26.164304 27055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:26.164400 27053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:26.164490 27047 server_base.cc:1034] running on GCE node
I20260927 15:11:26.165186 27047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:26.165881 27047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:26.167047 27047 hybrid_clock.cc:648] HybridClock initialized: now 1790521886167015 us; error 48 us; skew 500 ppm
I20260927 15:11:26.169126 27047 webserver.cc:460] Webserver started at http://127.7.17.193:46451/ using document root <none> and password file <none>
I20260927 15:11:26.169760 27047 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:26.169837 27047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:26.170053 27047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:26.172263 27047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-0/data/instance:
uuid: "a136b0bcac2146e5927b116e1d931c0a"
format_stamp: "Formatted at 2026-09-27 15:11:26 on dist-test-slave-hhpq"
server_key: "4abc8c1c3dc8f72380955a7ae53bc520"
server_key_iv: "701eae71f840a1a0b04a5d41bb086175"
server_key_version: "encryptionkey@0"
I20260927 15:11:26.172909 27047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance:
uuid: "a136b0bcac2146e5927b116e1d931c0a"
format_stamp: "Formatted at 2026-09-27 15:11:26 on dist-test-slave-hhpq"
server_key: "4abc8c1c3dc8f72380955a7ae53bc520"
server_key_iv: "701eae71f840a1a0b04a5d41bb086175"
server_key_version: "encryptionkey@0"
I20260927 15:11:26.177177 27047 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:11:26.179943 27061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:26.181485 27047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:26.181634 27047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-0/wal
uuid: "a136b0bcac2146e5927b116e1d931c0a"
format_stamp: "Formatted at 2026-09-27 15:11:26 on dist-test-slave-hhpq"
server_key: "4abc8c1c3dc8f72380955a7ae53bc520"
server_key_iv: "701eae71f840a1a0b04a5d41bb086175"
server_key_version: "encryptionkey@0"
I20260927 15:11:26.181756 27047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:26.202600 27047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:26.203414 27047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:26.203619 27047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:26.204298 27047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:26.205350 27047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:26.205404 27047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:26.205449 27047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:26.205463 27047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:26.214287 27047 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.193:37807
I20260927 15:11:26.214313 27174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.193:37807 every 8 connection(s)
I20260927 15:11:26.215257 27047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-0/data/info.pb
I20260927 15:11:26.224407 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 27047
I20260927 15:11:26.224535 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-0/wal/instance
I20260927 15:11:26.224915 7239 external_mini_cluster.cc:1428] Setting key 6096a63617e2dd09aabf7050cf11ef0a
I20260927 15:11:26.224972 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:26.217109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.193:49237 (local address 127.7.17.254:46527)
0927 15:11:26.217254 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:26.217258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:26.218446 (+ 1188us) server_negotiation.cc:407] Connection header received
0927 15:11:26.219838 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:26.219841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:26.219908 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:26.219985 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:26.220951 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.221506 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.222828 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.223074 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.223180 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:26.224405 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:26.224408 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:26.224410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:26.224425 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:26.224444 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:26.224451 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:26.224513 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:26.224707 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 15:11:26.224792 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:11:26.225957 27177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:26.217609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46527 (local address 127.7.17.193:49237)
0927 15:11:26.218290 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:26.218333 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:26.219617 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:26.220175 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:26.220183 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:26.220273 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:26.220782 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:26.220795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.221634 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.221637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:26.222683 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:26.222695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.223236 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.223240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:26.223438 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:26.224035 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:11:26.224037 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:26.224132 (+ 95us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:11:26.224233 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:26.224524 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:26.224534 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:26.224541 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:11:26.224675 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 15:11:26.224990 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":110,"threads_started":1}
I20260927 15:11:26.227012 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.194:0
--local_ip_for_outbound_sockets=127.7.17.194
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46527
--builtin_ntp_servers=127.7.17.212:44121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:11:26.227298 27175 heartbeater.cc:347] Connected to a master server at 127.7.17.254:46527
I20260927 15:11:26.227541 27175 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:26.228114 27175 heartbeater.cc:511] Master 127.7.17.254:46527 requested a full tablet report, sending...
I20260927 15:11:26.229738 26987 ts_manager.cc:194] Registered new tserver with Master: a136b0bcac2146e5927b116e1d931c0a (127.7.17.193:37807)
I20260927 15:11:26.230883 26987 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.17.193:49237
W20260927 15:11:26.339838 27178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:26.340142 27178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:26.340181 27178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:26.344630 27178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:26.344707 27178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:26.344767 27178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.194
I20260927 15:11:26.349820 27178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:44121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.17.194
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.194
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:26.350973 27178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:26.352367 27178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:26.364446 27183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:26.364576 27178 server_base.cc:1034] running on GCE node
W20260927 15:11:26.364588 27184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:26.364445 27186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:26.365372 27178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:26.366129 27178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:26.367307 27178 hybrid_clock.cc:648] HybridClock initialized: now 1790521886367282 us; error 34 us; skew 500 ppm
I20260927 15:11:26.369282 27178 webserver.cc:460] Webserver started at http://127.7.17.194:45131/ using document root <none> and password file <none>
I20260927 15:11:26.369866 27178 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:26.369928 27178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:26.370131 27178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:26.372213 27178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-1/data/instance:
uuid: "1191890f93ee4c79baa16774e9f3b77e"
format_stamp: "Formatted at 2026-09-27 15:11:26 on dist-test-slave-hhpq"
server_key: "677d6adfd2d26a5e7efb8778db727f1a"
server_key_iv: "060ccd3207f022acc230e5f55669d408"
server_key_version: "encryptionkey@0"
I20260927 15:11:26.372757 27178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance:
uuid: "1191890f93ee4c79baa16774e9f3b77e"
format_stamp: "Formatted at 2026-09-27 15:11:26 on dist-test-slave-hhpq"
server_key: "677d6adfd2d26a5e7efb8778db727f1a"
server_key_iv: "060ccd3207f022acc230e5f55669d408"
server_key_version: "encryptionkey@0"
I20260927 15:11:26.376411 27178 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:11:26.378975 27192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:26.380188 27178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:11:26.380298 27178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-1/wal
uuid: "1191890f93ee4c79baa16774e9f3b77e"
format_stamp: "Formatted at 2026-09-27 15:11:26 on dist-test-slave-hhpq"
server_key: "677d6adfd2d26a5e7efb8778db727f1a"
server_key_iv: "060ccd3207f022acc230e5f55669d408"
server_key_version: "encryptionkey@0"
I20260927 15:11:26.380419 27178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:26.400586 27178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:26.401396 27178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:26.401595 27178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:26.402220 27178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:26.403226 27178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:26.403276 27178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:26.403329 27178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:26.403358 27178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:26.413564 27178 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.194:43061
I20260927 15:11:26.413622 27305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.194:43061 every 8 connection(s)
I20260927 15:11:26.414852 27178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-1/data/info.pb
I20260927 15:11:26.423832 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 27178
I20260927 15:11:26.423954 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-1/wal/instance
I20260927 15:11:26.424232 7239 external_mini_cluster.cc:1428] Setting key 4d5740f5f8f8407454d1ad52f1585530
I20260927 15:11:26.424798 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:26.416947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:39209 (local address 127.7.17.254:46527)
0927 15:11:26.417093 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:26.417097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:26.418369 (+ 1272us) server_negotiation.cc:407] Connection header received
0927 15:11:26.419880 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:26.419884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:26.419938 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:26.420034 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:26.421095 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.421653 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.422803 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.423025 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.423108 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:26.424304 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:26.424308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:26.424311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:26.424329 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:26.424352 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:26.424363 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:26.424439 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:26.424598 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 15:11:26.424675 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:11:26.425860 27308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:26.417495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46527 (local address 127.7.17.194:39209)
0927 15:11:26.418203 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:26.418249 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:26.419597 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:26.420209 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:26.420217 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:26.420318 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:26.420940 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:26.420959 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.421761 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.421764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:26.422694 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:26.422704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.423153 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.423155 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:26.423305 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:26.423971 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:11:26.423974 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:26.424040 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:11:26.424133 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:26.424457 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:26.424466 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:26.424471 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:11:26.424579 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 15:11:26.424933 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":112,"threads_started":1}
I20260927 15:11:26.426339 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.17.195:0
--local_ip_for_outbound_sockets=127.7.17.195
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46527
--builtin_ntp_servers=127.7.17.212:44121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:11:26.427109 27306 heartbeater.cc:347] Connected to a master server at 127.7.17.254:46527
I20260927 15:11:26.427349 27306 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:26.428032 27306 heartbeater.cc:511] Master 127.7.17.254:46527 requested a full tablet report, sending...
I20260927 15:11:26.429257 26987 ts_manager.cc:194] Registered new tserver with Master: 1191890f93ee4c79baa16774e9f3b77e (127.7.17.194:43061)
I20260927 15:11:26.430006 26987 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.17.194:39209
W20260927 15:11:26.541034 27309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:26.541347 27309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:26.541389 27309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:26.545532 27309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:26.545603 27309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:26.545660 27309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.17.195
I20260927 15:11:26.550510 27309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:44121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.17.195
--webserver_port=0
--tserver_master_addrs=127.7.17.254:46527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.17.195
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:26.551616 27309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:26.553088 27309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:26.563027 27315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:26.563017 27317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:26.563033 27314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:26.563108 27309 server_base.cc:1034] running on GCE node
I20260927 15:11:26.563669 27309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:26.564359 27309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:26.565546 27309 hybrid_clock.cc:648] HybridClock initialized: now 1790521886565513 us; error 46 us; skew 500 ppm
I20260927 15:11:26.567623 27309 webserver.cc:460] Webserver started at http://127.7.17.195:45881/ using document root <none> and password file <none>
I20260927 15:11:26.568228 27309 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:26.568298 27309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:26.568491 27309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:26.570763 27309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-2/data/instance:
uuid: "0d9a107094c34553be59df449a68e825"
format_stamp: "Formatted at 2026-09-27 15:11:26 on dist-test-slave-hhpq"
server_key: "b30f274b4fb4e5ac2acc58fa0fdf6b40"
server_key_iv: "1feed7891b448a643357335dbc8cd9cf"
server_key_version: "encryptionkey@0"
I20260927 15:11:26.571369 27309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance:
uuid: "0d9a107094c34553be59df449a68e825"
format_stamp: "Formatted at 2026-09-27 15:11:26 on dist-test-slave-hhpq"
server_key: "b30f274b4fb4e5ac2acc58fa0fdf6b40"
server_key_iv: "1feed7891b448a643357335dbc8cd9cf"
server_key_version: "encryptionkey@0"
I20260927 15:11:26.575388 27309 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 15:11:26.578204 27323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:26.579679 27309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:11:26.579829 27309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-2/wal
uuid: "0d9a107094c34553be59df449a68e825"
format_stamp: "Formatted at 2026-09-27 15:11:26 on dist-test-slave-hhpq"
server_key: "b30f274b4fb4e5ac2acc58fa0fdf6b40"
server_key_iv: "1feed7891b448a643357335dbc8cd9cf"
server_key_version: "encryptionkey@0"
I20260927 15:11:26.579947 27309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:26.598872 27309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:26.599633 27309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:26.599818 27309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:26.600448 27309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:11:26.601485 27309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:11:26.601534 27309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:26.601591 27309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:11:26.601619 27309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:26.610348 27309 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.195:35865
I20260927 15:11:26.610416 27436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.195:35865 every 8 connection(s)
I20260927 15:11:26.611325 27309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-2/data/info.pb
I20260927 15:11:26.613240 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 27309
I20260927 15:11:26.613354 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-2/wal/instance
I20260927 15:11:26.613636 7239 external_mini_cluster.cc:1428] Setting key 99250d61659ecf8600e672d025f5416a
I20260927 15:11:26.621006 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:26.613296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.195:54531 (local address 127.7.17.254:46527)
0927 15:11:26.613453 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:26.613457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:26.614682 (+ 1225us) server_negotiation.cc:407] Connection header received
0927 15:11:26.616206 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:26.616209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:26.616266 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:26.616338 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:26.617320 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.617862 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.619032 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.619261 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.619350 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:26.620507 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:26.620511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:26.620513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:26.620526 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:26.620544 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:26.620552 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:26.620610 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:26.620762 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 15:11:26.620854 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 15:11:26.621896 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:26.613751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:46527 (local address 127.7.17.195:54531)
0927 15:11:26.614538 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:26.614601 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:26.616003 (+ 1402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:26.616517 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:26.616525 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:26.616617 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:26.617185 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:26.617199 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.617996 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.617999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:26.618913 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:26.618923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.619410 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.619415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:26.619578 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:26.620214 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:11:26.620216 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:26.620264 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:11:26.620340 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:26.620629 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:26.620638 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:26.620643 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:11:26.620753 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 15:11:26.621101 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":125,"threads_started":1}
I20260927 15:11:26.623129 27437 heartbeater.cc:347] Connected to a master server at 127.7.17.254:46527
I20260927 15:11:26.623380 27437 heartbeater.cc:464] Registering TS with master...
I20260927 15:11:26.623905 27437 heartbeater.cc:511] Master 127.7.17.254:46527 requested a full tablet report, sending...
I20260927 15:11:26.624985 26987 ts_manager.cc:194] Registered new tserver with Master: 0d9a107094c34553be59df449a68e825 (127.7.17.195:35865)
I20260927 15:11:26.625556 26987 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.17.195:54531
I20260927 15:11:26.627713 7239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:11:26.633040 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:26.629553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43826 (local address 127.7.17.254:46527)
0927 15:11:26.629725 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:26.629730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:26.629833 (+ 103us) server_negotiation.cc:407] Connection header received
0927 15:11:26.629930 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:26.629933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:26.629971 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:26.630033 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:26.630257 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.630774 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.631559 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.631794 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.631875 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:26.632630 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:26.632633 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:26.632635 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:26.632647 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:26.632676 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:26.632683 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:26.632745 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:26.632814 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 15:11:26.632898 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 15:11:26.636124 26987 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:11:26.639173 26987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:11:26.653317 27448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:26.647730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:43061 (local address 127.0.0.1:41996)
0927 15:11:26.648055 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:26.648104 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:26.648226 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:26.649507 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:26.649513 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:26.649534 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:26.649694 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:26.649703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.650775 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.650778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:26.651886 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:26.651895 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.652711 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.652714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:26.652855 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:26.652903 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 15:11:26.653022 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 15:11:26.653144 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":34,"thread_start_us":78,"threads_started":1}
I20260927 15:11:26.653338 27451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:26.648407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41996 (local address 127.7.17.194:43061)
0927 15:11:26.649082 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:26.649092 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:26.649115 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:11:26.649187 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:26.649191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:26.649364 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:26.649502 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:26.649827 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.650661 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.652014 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.652594 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.652696 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:26.652932 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:26.653061 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 15:11:26.653173 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":132,"threads_started":1}
I20260927 15:11:26.654134 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:26.647805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:37807 (local address 127.0.0.1:33490)
0927 15:11:26.648606 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:26.648619 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:26.648713 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:26.649740 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:26.649743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:26.649762 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:26.649837 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:26.649842 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.651281 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.651283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:26.652436 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:26.652446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.653753 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.653756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:26.653843 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:26.653865 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:11:26.653936 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 15:11:26.654004 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":116,"threads_started":1}
I20260927 15:11:26.654394 27453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:26.648851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33490 (local address 127.7.17.193:37807)
0927 15:11:26.649350 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:26.649360 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:26.649380 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:11:26.649451 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:26.649455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:26.649609 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:26.649749 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:26.649950 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.651127 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.652573 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.653401 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.653499 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:26.653690 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:26.654005 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 15:11:26.654107 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":89,"threads_started":1}
I20260927 15:11:26.654827 27449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:26.648019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:35865 (local address 127.0.0.1:58108)
0927 15:11:26.648897 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:26.648908 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:26.648986 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:26.649682 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:26.649685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:26.649703 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:26.649804 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:26.649810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.651530 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.651533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:26.652627 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:26.652634 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.654406 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.654410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:26.654528 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:26.654551 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:11:26.654623 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 15:11:26.654703 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260927 15:11:26.655328 27452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:26.648507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58108 (local address 127.7.17.195:35865)
0927 15:11:26.648893 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:26.648905 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:26.649074 (+ 169us) server_negotiation.cc:407] Connection header received
0927 15:11:26.649158 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:26.649165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:26.649381 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:26.649521 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:26.649929 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.651127 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.652757 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.653891 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.653991 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:26.654167 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:26.655071 (+ 904us) server_negotiation.cc:299] Negotiation successful
0927 15:11:26.655177 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":93,"threads_started":1}
I20260927 15:11:26.656567 27240 tablet_service.cc:1467] Processing CreateTablet for tablet 2814ed44440345269d78e933388d5616 (DEFAULT_TABLE table=test-table [id=e6a2a6508f994a5f8c16e9385f142546]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:26.657233 27109 tablet_service.cc:1467] Processing CreateTablet for tablet 2814ed44440345269d78e933388d5616 (DEFAULT_TABLE table=test-table [id=e6a2a6508f994a5f8c16e9385f142546]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:26.657796 27240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2814ed44440345269d78e933388d5616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:26.657747 27371 tablet_service.cc:1467] Processing CreateTablet for tablet 2814ed44440345269d78e933388d5616 (DEFAULT_TABLE table=test-table [id=e6a2a6508f994a5f8c16e9385f142546]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:11:26.658281 27109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2814ed44440345269d78e933388d5616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:26.658615 27371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2814ed44440345269d78e933388d5616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:26.664644 27454 tablet_bootstrap.cc:492] T 2814ed44440345269d78e933388d5616 P 0d9a107094c34553be59df449a68e825: Bootstrap starting.
I20260927 15:11:26.666153 27455 tablet_bootstrap.cc:492] T 2814ed44440345269d78e933388d5616 P a136b0bcac2146e5927b116e1d931c0a: Bootstrap starting.
I20260927 15:11:26.667668 27454 tablet_bootstrap.cc:654] T 2814ed44440345269d78e933388d5616 P 0d9a107094c34553be59df449a68e825: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:26.667971 27455 tablet_bootstrap.cc:654] T 2814ed44440345269d78e933388d5616 P a136b0bcac2146e5927b116e1d931c0a: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:26.668426 27454 log.cc:824] T 2814ed44440345269d78e933388d5616 P 0d9a107094c34553be59df449a68e825: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:26.668629 27455 log.cc:824] T 2814ed44440345269d78e933388d5616 P a136b0bcac2146e5927b116e1d931c0a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:26.669188 27456 tablet_bootstrap.cc:492] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e: Bootstrap starting.
I20260927 15:11:26.670630 27454 tablet_bootstrap.cc:492] T 2814ed44440345269d78e933388d5616 P 0d9a107094c34553be59df449a68e825: No bootstrap required, opened a new log
I20260927 15:11:26.670630 27455 tablet_bootstrap.cc:492] T 2814ed44440345269d78e933388d5616 P a136b0bcac2146e5927b116e1d931c0a: No bootstrap required, opened a new log
I20260927 15:11:26.670835 27455 ts_tablet_manager.cc:1397] T 2814ed44440345269d78e933388d5616 P a136b0bcac2146e5927b116e1d931c0a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 15:11:26.670835 27454 ts_tablet_manager.cc:1397] T 2814ed44440345269d78e933388d5616 P 0d9a107094c34553be59df449a68e825: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:11:26.671129 27456 tablet_bootstrap.cc:654] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:26.671949 27456 log.cc:824] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:26.673811 27456 tablet_bootstrap.cc:492] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e: No bootstrap required, opened a new log
I20260927 15:11:26.674005 27456 ts_tablet_manager.cc:1397] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 15:11:26.675212 27454 raft_consensus.cc:348] T 2814ed44440345269d78e933388d5616 P 0d9a107094c34553be59df449a68e825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1191890f93ee4c79baa16774e9f3b77e" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 43061 } } peers { permanent_uuid: "a136b0bcac2146e5927b116e1d931c0a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37807 } } peers { permanent_uuid: "0d9a107094c34553be59df449a68e825" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35865 } }
I20260927 15:11:26.675212 27455 raft_consensus.cc:348] T 2814ed44440345269d78e933388d5616 P a136b0bcac2146e5927b116e1d931c0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1191890f93ee4c79baa16774e9f3b77e" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 43061 } } peers { permanent_uuid: "a136b0bcac2146e5927b116e1d931c0a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37807 } } peers { permanent_uuid: "0d9a107094c34553be59df449a68e825" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35865 } }
I20260927 15:11:26.675505 27455 raft_consensus.cc:374] T 2814ed44440345269d78e933388d5616 P a136b0bcac2146e5927b116e1d931c0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:26.675505 27454 raft_consensus.cc:374] T 2814ed44440345269d78e933388d5616 P 0d9a107094c34553be59df449a68e825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:26.675565 27454 raft_consensus.cc:729] T 2814ed44440345269d78e933388d5616 P 0d9a107094c34553be59df449a68e825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9a107094c34553be59df449a68e825, State: Initialized, Role: FOLLOWER
I20260927 15:11:26.675573 27455 raft_consensus.cc:729] T 2814ed44440345269d78e933388d5616 P a136b0bcac2146e5927b116e1d931c0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a136b0bcac2146e5927b116e1d931c0a, State: Initialized, Role: FOLLOWER
I20260927 15:11:26.676059 27454 consensus_queue.cc:260] T 2814ed44440345269d78e933388d5616 P 0d9a107094c34553be59df449a68e825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1191890f93ee4c79baa16774e9f3b77e" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 43061 } } peers { permanent_uuid: "a136b0bcac2146e5927b116e1d931c0a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37807 } } peers { permanent_uuid: "0d9a107094c34553be59df449a68e825" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35865 } }
I20260927 15:11:26.676059 27455 consensus_queue.cc:260] T 2814ed44440345269d78e933388d5616 P a136b0bcac2146e5927b116e1d931c0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1191890f93ee4c79baa16774e9f3b77e" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 43061 } } peers { permanent_uuid: "a136b0bcac2146e5927b116e1d931c0a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37807 } } peers { permanent_uuid: "0d9a107094c34553be59df449a68e825" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35865 } }
I20260927 15:11:26.677052 27437 heartbeater.cc:503] Master 127.7.17.254:46527 was elected leader, sending a full tablet report...
I20260927 15:11:26.677176 27454 ts_tablet_manager.cc:1428] T 2814ed44440345269d78e933388d5616 P 0d9a107094c34553be59df449a68e825: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260927 15:11:26.677176 27455 ts_tablet_manager.cc:1428] T 2814ed44440345269d78e933388d5616 P a136b0bcac2146e5927b116e1d931c0a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 15:11:26.677680 27456 raft_consensus.cc:348] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1191890f93ee4c79baa16774e9f3b77e" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 43061 } } peers { permanent_uuid: "a136b0bcac2146e5927b116e1d931c0a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37807 } } peers { permanent_uuid: "0d9a107094c34553be59df449a68e825" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35865 } }
I20260927 15:11:26.677930 27456 raft_consensus.cc:374] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:26.677975 27456 raft_consensus.cc:729] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1191890f93ee4c79baa16774e9f3b77e, State: Initialized, Role: FOLLOWER
I20260927 15:11:26.678251 27175 heartbeater.cc:503] Master 127.7.17.254:46527 was elected leader, sending a full tablet report...
I20260927 15:11:26.678632 27456 consensus_queue.cc:260] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1191890f93ee4c79baa16774e9f3b77e" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 43061 } } peers { permanent_uuid: "a136b0bcac2146e5927b116e1d931c0a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37807 } } peers { permanent_uuid: "0d9a107094c34553be59df449a68e825" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35865 } }
I20260927 15:11:26.679663 27456 ts_tablet_manager.cc:1428] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:11:26.679788 27306 heartbeater.cc:503] Master 127.7.17.254:46527 was elected leader, sending a full tablet report...
W20260927 15:11:26.716444 27176 tablet.cc:2369] T 2814ed44440345269d78e933388d5616 P a136b0bcac2146e5927b116e1d931c0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:11:26.732527 27462 raft_consensus.cc:482] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:11:26.732748 27462 raft_consensus.cc:504] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1191890f93ee4c79baa16774e9f3b77e" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 43061 } } peers { permanent_uuid: "a136b0bcac2146e5927b116e1d931c0a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37807 } } peers { permanent_uuid: "0d9a107094c34553be59df449a68e825" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35865 } }
I20260927 15:11:26.733919 27462 leader_election.cc:288] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a136b0bcac2146e5927b116e1d931c0a (127.7.17.193:37807), 0d9a107094c34553be59df449a68e825 (127.7.17.195:35865)
I20260927 15:11:26.737816 27452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:26.734302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:57911 (local address 127.7.17.195:35865)
0927 15:11:26.734433 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:26.734438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:26.734769 (+ 331us) server_negotiation.cc:407] Connection header received
0927 15:11:26.734910 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:26.734913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:26.734975 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:26.735047 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:26.735300 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.735804 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.736813 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.737249 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.737359 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:26.737411 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:26.737621 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 15:11:26.737693 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:11:26.737815 27453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:26.734393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.17.194:44867 (local address 127.7.17.193:37807)
0927 15:11:26.734505 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:26.734509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:26.734522 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:11:26.734581 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:26.734584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:26.734637 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:26.734728 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:11:26.735054 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.735652 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.736819 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.737242 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.737354 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:26.737413 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:26.737610 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 15:11:26.737685 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:11:26.737818 27308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:26.734185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:37807 (local address 127.7.17.194:44867)
0927 15:11:26.734320 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:26.734336 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:26.734439 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:26.734744 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:26.734748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:26.734795 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:26.734921 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:26.734929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.735766 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.735770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:26.736662 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:26.736693 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.737359 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.737363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:26.737488 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:26.737511 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:11:26.737573 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 15:11:26.737653 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":6,"spinlock_wait_cycles":33920}
I20260927 15:11:26.737835 27463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:26.734241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.195:35865 (local address 127.7.17.194:57911)
0927 15:11:26.734651 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:26.734663 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:26.734804 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:26.735060 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:26.735063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:26.735080 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:11:26.735201 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:26.735206 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.735906 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.735908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:26.736682 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:26.736692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.737356 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.737359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:26.737499 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:26.737517 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 15:11:26.737593 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 15:11:26.737662 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":24,"spinlock_wait_cycles":7936,"thread_start_us":140,"threads_started":1}
I20260927 15:11:26.738590 27129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2814ed44440345269d78e933388d5616" candidate_uuid: "1191890f93ee4c79baa16774e9f3b77e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a136b0bcac2146e5927b116e1d931c0a" is_pre_election: true
I20260927 15:11:26.738595 27391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2814ed44440345269d78e933388d5616" candidate_uuid: "1191890f93ee4c79baa16774e9f3b77e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9a107094c34553be59df449a68e825" is_pre_election: true
I20260927 15:11:26.738940 27129 raft_consensus.cc:2454] T 2814ed44440345269d78e933388d5616 P a136b0bcac2146e5927b116e1d931c0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1191890f93ee4c79baa16774e9f3b77e in term 0.
I20260927 15:11:26.738940 27391 raft_consensus.cc:2454] T 2814ed44440345269d78e933388d5616 P 0d9a107094c34553be59df449a68e825 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1191890f93ee4c79baa16774e9f3b77e in term 0.
I20260927 15:11:26.739492 27195 leader_election.cc:302] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9a107094c34553be59df449a68e825, 1191890f93ee4c79baa16774e9f3b77e; no voters:
I20260927 15:11:26.739807 27462 raft_consensus.cc:2781] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:11:26.739890 27462 raft_consensus.cc:482] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:11:26.739934 27462 raft_consensus.cc:3037] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:26.740988 27462 raft_consensus.cc:504] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1191890f93ee4c79baa16774e9f3b77e" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 43061 } } peers { permanent_uuid: "a136b0bcac2146e5927b116e1d931c0a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37807 } } peers { permanent_uuid: "0d9a107094c34553be59df449a68e825" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35865 } }
I20260927 15:11:26.741356 27462 leader_election.cc:288] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e [CANDIDATE]: Term 1 election: Requested vote from peers a136b0bcac2146e5927b116e1d931c0a (127.7.17.193:37807), 0d9a107094c34553be59df449a68e825 (127.7.17.195:35865)
I20260927 15:11:26.741786 27129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2814ed44440345269d78e933388d5616" candidate_uuid: "1191890f93ee4c79baa16774e9f3b77e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a136b0bcac2146e5927b116e1d931c0a"
I20260927 15:11:26.741786 27391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2814ed44440345269d78e933388d5616" candidate_uuid: "1191890f93ee4c79baa16774e9f3b77e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9a107094c34553be59df449a68e825"
I20260927 15:11:26.741927 27129 raft_consensus.cc:3037] T 2814ed44440345269d78e933388d5616 P a136b0bcac2146e5927b116e1d931c0a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:26.741927 27391 raft_consensus.cc:3037] T 2814ed44440345269d78e933388d5616 P 0d9a107094c34553be59df449a68e825 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:26.743263 27391 raft_consensus.cc:2454] T 2814ed44440345269d78e933388d5616 P 0d9a107094c34553be59df449a68e825 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1191890f93ee4c79baa16774e9f3b77e in term 1.
I20260927 15:11:26.743263 27129 raft_consensus.cc:2454] T 2814ed44440345269d78e933388d5616 P a136b0bcac2146e5927b116e1d931c0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1191890f93ee4c79baa16774e9f3b77e in term 1.
I20260927 15:11:26.743603 27195 leader_election.cc:302] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0d9a107094c34553be59df449a68e825, 1191890f93ee4c79baa16774e9f3b77e, a136b0bcac2146e5927b116e1d931c0a; no voters:
I20260927 15:11:26.743803 27462 raft_consensus.cc:2781] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:26.744064 27462 raft_consensus.cc:686] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e [term 1 LEADER]: Becoming Leader. State: Replica: 1191890f93ee4c79baa16774e9f3b77e, State: Running, Role: LEADER
I20260927 15:11:26.744403 27462 consensus_queue.cc:237] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1191890f93ee4c79baa16774e9f3b77e" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 43061 } } peers { permanent_uuid: "a136b0bcac2146e5927b116e1d931c0a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37807 } } peers { permanent_uuid: "0d9a107094c34553be59df449a68e825" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35865 } }
I20260927 15:11:26.747560 26985 catalog_manager.cc:5495] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1191890f93ee4c79baa16774e9f3b77e (127.7.17.194). New cstate: current_term: 1 leader_uuid: "1191890f93ee4c79baa16774e9f3b77e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1191890f93ee4c79baa16774e9f3b77e" member_type: VOTER last_known_addr { host: "127.7.17.194" port: 43061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a136b0bcac2146e5927b116e1d931c0a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d9a107094c34553be59df449a68e825" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35865 } health_report { overall_health: UNKNOWN } } }
I20260927 15:11:26.779726 27451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:26.776129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41998 (local address 127.7.17.194:43061)
0927 15:11:26.776279 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:26.776283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:26.776299 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:11:26.776391 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:26.776394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:26.776453 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:26.776552 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:11:26.776803 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.777366 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.778267 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:26.778484 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:26.778575 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:26.778877 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:11:26.779336 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:11:26.779453 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:26.779515 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 15:11:26.779577 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:11:26.786428 27129 raft_consensus.cc:1261] T 2814ed44440345269d78e933388d5616 P a136b0bcac2146e5927b116e1d931c0a [term 1 FOLLOWER]: Refusing update from remote peer 1191890f93ee4c79baa16774e9f3b77e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:26.786428 27391 raft_consensus.cc:1261] T 2814ed44440345269d78e933388d5616 P 0d9a107094c34553be59df449a68e825 [term 1 FOLLOWER]: Refusing update from remote peer 1191890f93ee4c79baa16774e9f3b77e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:26.787341 27464 consensus_queue.cc:1035] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9a107094c34553be59df449a68e825" member_type: VOTER last_known_addr { host: "127.7.17.195" port: 35865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:26.787557 27462 consensus_queue.cc:1035] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a136b0bcac2146e5927b116e1d931c0a" member_type: VOTER last_known_addr { host: "127.7.17.193" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:26.797197 27468 mvcc.cc:205] Tried to move back new op lower bound from 7333977648268681216 to 7333977648107962368. Current Snapshot: MvccSnapshot[applied={T|T < 7333977648268681216}]
I20260927 15:11:26.803822 26987 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:26.803994 26987 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:11:26.806548 26987 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 073e668a7d7348389b4942eb5370366f: Sending DeleteTablet for 3 replicas of tablet 2814ed44440345269d78e933388d5616
I20260927 15:11:26.807390 27240 tablet_service.cc:1514] Processing DeleteTablet for tablet 2814ed44440345269d78e933388d5616 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:26 UTC) from {username='slave'} at 127.0.0.1:41996
I20260927 15:11:26.807421 27109 tablet_service.cc:1514] Processing DeleteTablet for tablet 2814ed44440345269d78e933388d5616 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:26 UTC) from {username='slave'} at 127.0.0.1:33490
I20260927 15:11:26.807492 27371 tablet_service.cc:1514] Processing DeleteTablet for tablet 2814ed44440345269d78e933388d5616 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:11:26 UTC) from {username='slave'} at 127.0.0.1:58108
I20260927 15:11:26.807977 27477 tablet_replica.cc:330] T 2814ed44440345269d78e933388d5616 P a136b0bcac2146e5927b116e1d931c0a: stopping tablet replica
I20260927 15:11:26.807988 27479 tablet_replica.cc:330] T 2814ed44440345269d78e933388d5616 P 0d9a107094c34553be59df449a68e825: stopping tablet replica
I20260927 15:11:26.808349 27479 raft_consensus.cc:2229] T 2814ed44440345269d78e933388d5616 P 0d9a107094c34553be59df449a68e825 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:11:26.808362 27477 raft_consensus.cc:2229] T 2814ed44440345269d78e933388d5616 P a136b0bcac2146e5927b116e1d931c0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:11:26.808503 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 27047
I20260927 15:11:26.808701 27477 raft_consensus.cc:2258] T 2814ed44440345269d78e933388d5616 P a136b0bcac2146e5927b116e1d931c0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:26.809724 27479 raft_consensus.cc:2258] T 2814ed44440345269d78e933388d5616 P 0d9a107094c34553be59df449a68e825 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:11:26.811946 27479 ts_tablet_manager.cc:1905] T 2814ed44440345269d78e933388d5616 P 0d9a107094c34553be59df449a68e825: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:11:26.814813 27479 ts_tablet_manager.cc:1918] T 2814ed44440345269d78e933388d5616 P 0d9a107094c34553be59df449a68e825: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:11:26.814916 27479 log.cc:1196] T 2814ed44440345269d78e933388d5616 P 0d9a107094c34553be59df449a68e825: Deleting WAL directory at /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521636012660-7239-0/minicluster-data/ts-2/wal/wals/2814ed44440345269d78e933388d5616
W20260927 15:11:26.815054 26972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.17.193:37807 (error 108)
I20260927 15:11:26.815213 27479 ts_tablet_manager.cc:1939] T 2814ed44440345269d78e933388d5616 P 0d9a107094c34553be59df449a68e825: Deleting consensus metadata
I20260927 15:11:26.815429 27478 tablet_replica.cc:330] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e: stopping tablet replica
I20260927 15:11:26.815837 27478 raft_consensus.cc:2229] T 2814ed44440345269d78e933388d5616 P 1191890f93ee4c79baa16774e9f3b77e [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:11:26.816474 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 27178
I20260927 15:11:26.816520 27449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:26.816238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.193:37807 (local address 127.0.0.1:33496)
0927 15:11:26.816357 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:26.816429 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.193:37807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 15:11:26.816540 26973 catalog_manager.cc:4841] TS 0d9a107094c34553be59df449a68e825 (127.7.17.195:35865): tablet 2814ed44440345269d78e933388d5616 (table test-table [id=e6a2a6508f994a5f8c16e9385f142546]) successfully deleted
W20260927 15:11:26.817910 26972 catalog_manager.cc:4584] TS a136b0bcac2146e5927b116e1d931c0a (127.7.17.193:37807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2814ed44440345269d78e933388d5616: Network error: Client connection negotiation failed: client connection to 127.7.17.193:37807: connect: Connection refused (error 111)
I20260927 15:11:26.821579 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 27309
I20260927 15:11:26.822077 27449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:26.821771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.194:43061 (local address 127.0.0.1:42006)
0927 15:11:26.821938 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:26.821994 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.194:43061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 15:11:26.826349 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 26956
2026-09-27T15:11:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1068 ms)
[----------] 6 tests from EncryptionPolicyTest (10085 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:26 dist-test-slave-hhpq krb5kdc[27483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:26 dist-test-slave-hhpq krb5kdc[27483](info): set up 2 sockets
Sep 27 15:11:26 dist-test-slave-hhpq krb5kdc[27483](info): commencing operation
krb5kdc: starting...
W20260927 15:11:27.796865 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:11:27 dist-test-slave-hhpq krb5kdc[27483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:11:27.919967 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36825
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:28.030094 27496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:28.030385 27496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:28.030453 27496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:28.034529 27496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:28.034605 27496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:28.034631 27496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:28.034654 27496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:28.034684 27496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:28.039990 27496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36825
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:28.041139 27496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:28.042138 27496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:28.050664 27502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:28.050812 27496 server_base.cc:1034] running on GCE node
W20260927 15:11:28.050711 27501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:28.050643 27504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:28.052073 27496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:28.053519 27496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:28.054704 27496 hybrid_clock.cc:648] HybridClock initialized: now 1790521888054637 us; error 86 us; skew 500 ppm
Sep 27 15:11:28 dist-test-slave-hhpq krb5kdc[27483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:28.057570 27496 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:11:28.058561 27496 webserver.cc:460] Webserver started at http://127.0.0.1:42061/ using document root <none> and password file <none>
I20260927 15:11:28.059118 27496 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:28.059183 27496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:28.059355 27496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:28.061672 27496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "e51ba110882f4faa9394058c43e1f5a7"
format_stamp: "Formatted at 2026-09-27 15:11:28 on dist-test-slave-hhpq"
server_key: "dca257561b4782d326f9d570ba057db3"
server_key_iv: "b11abe594e1c6e0919403f05b2ffd927"
server_key_version: "encryptionkey@0"
I20260927 15:11:28.062268 27496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "e51ba110882f4faa9394058c43e1f5a7"
format_stamp: "Formatted at 2026-09-27 15:11:28 on dist-test-slave-hhpq"
server_key: "dca257561b4782d326f9d570ba057db3"
server_key_iv: "b11abe594e1c6e0919403f05b2ffd927"
server_key_version: "encryptionkey@0"
I20260927 15:11:28.066483 27496 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:11:28.069154 27511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:28.070467 27496 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:11:28.070604 27496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "e51ba110882f4faa9394058c43e1f5a7"
format_stamp: "Formatted at 2026-09-27 15:11:28 on dist-test-slave-hhpq"
server_key: "dca257561b4782d326f9d570ba057db3"
server_key_iv: "b11abe594e1c6e0919403f05b2ffd927"
server_key_version: "encryptionkey@0"
I20260927 15:11:28.070719 27496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:28.087018 27496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:28.089996 27496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:28.090183 27496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:28.096745 27496 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36825
I20260927 15:11:28.096786 27563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36825 every 8 connection(s)
I20260927 15:11:28.097741 27496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:28.100634 27564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:28.106541 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 27496
I20260927 15:11:28.106664 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:28.106719 27564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e51ba110882f4faa9394058c43e1f5a7: Bootstrap starting.
I20260927 15:11:28.106950 7239 external_mini_cluster.cc:1428] Setting key f6887d7c316da8f90cd3ff5a902f5799
I20260927 15:11:28.109469 27564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e51ba110882f4faa9394058c43e1f5a7: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:28.110239 27564 log.cc:824] T 00000000000000000000000000000000 P e51ba110882f4faa9394058c43e1f5a7: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:28.112353 27564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e51ba110882f4faa9394058c43e1f5a7: No bootstrap required, opened a new log
Sep 27 15:11:28 dist-test-slave-hhpq krb5kdc[27483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:11:28.116816 27564 raft_consensus.cc:348] T 00000000000000000000000000000000 P e51ba110882f4faa9394058c43e1f5a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51ba110882f4faa9394058c43e1f5a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } }
I20260927 15:11:28.117152 27564 raft_consensus.cc:374] T 00000000000000000000000000000000 P e51ba110882f4faa9394058c43e1f5a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:28.117215 27564 raft_consensus.cc:729] T 00000000000000000000000000000000 P e51ba110882f4faa9394058c43e1f5a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e51ba110882f4faa9394058c43e1f5a7, State: Initialized, Role: FOLLOWER
I20260927 15:11:28.117695 27564 consensus_queue.cc:260] T 00000000000000000000000000000000 P e51ba110882f4faa9394058c43e1f5a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51ba110882f4faa9394058c43e1f5a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } }
I20260927 15:11:28.117858 27564 raft_consensus.cc:388] T 00000000000000000000000000000000 P e51ba110882f4faa9394058c43e1f5a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:28.117928 27564 raft_consensus.cc:482] T 00000000000000000000000000000000 P e51ba110882f4faa9394058c43e1f5a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:28.118027 27564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e51ba110882f4faa9394058c43e1f5a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:28.118924 27564 raft_consensus.cc:504] T 00000000000000000000000000000000 P e51ba110882f4faa9394058c43e1f5a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51ba110882f4faa9394058c43e1f5a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } }
I20260927 15:11:28.119289 27564 leader_election.cc:302] T 00000000000000000000000000000000 P e51ba110882f4faa9394058c43e1f5a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e51ba110882f4faa9394058c43e1f5a7; no voters:
I20260927 15:11:28.119699 27564 leader_election.cc:288] T 00000000000000000000000000000000 P e51ba110882f4faa9394058c43e1f5a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:28.119755 27569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e51ba110882f4faa9394058c43e1f5a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:28.119928 27569 raft_consensus.cc:686] T 00000000000000000000000000000000 P e51ba110882f4faa9394058c43e1f5a7 [term 1 LEADER]: Becoming Leader. State: Replica: e51ba110882f4faa9394058c43e1f5a7, State: Running, Role: LEADER
I20260927 15:11:28.120250 27569 consensus_queue.cc:237] T 00000000000000000000000000000000 P e51ba110882f4faa9394058c43e1f5a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51ba110882f4faa9394058c43e1f5a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } }
I20260927 15:11:28.121068 27564 sys_catalog.cc:564] T 00000000000000000000000000000000 P e51ba110882f4faa9394058c43e1f5a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:28.122673 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:28.108408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42965 (local address 127.0.0.1:36825)
0927 15:11:28.108870 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:28.108889 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:28.108927 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:11:28.109668 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:28.109694 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:28.110090 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:28.110435 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:28.111304 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.112350 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.113542 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.114019 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.114134 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:28.117112 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:28.117118 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:28.117132 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:28.117162 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:28.119043 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:28.119638 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:28.119644 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:28.119649 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:28.119770 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:28.120242 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:28.120245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:28.120247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:28.120778 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:28.121032 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:28.121365 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 15:11:28.121615 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":126,"threads_started":1}
I20260927 15:11:28.123018 27571 sys_catalog.cc:455] T 00000000000000000000000000000000 P e51ba110882f4faa9394058c43e1f5a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e51ba110882f4faa9394058c43e1f5a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51ba110882f4faa9394058c43e1f5a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } } }
I20260927 15:11:28.123138 27571 sys_catalog.cc:458] T 00000000000000000000000000000000 P e51ba110882f4faa9394058c43e1f5a7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:28.123826 27584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:28.123867 27570 sys_catalog.cc:455] T 00000000000000000000000000000000 P e51ba110882f4faa9394058c43e1f5a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e51ba110882f4faa9394058c43e1f5a7. Latest consensus state: current_term: 1 leader_uuid: "e51ba110882f4faa9394058c43e1f5a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51ba110882f4faa9394058c43e1f5a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } } }
I20260927 15:11:28.123942 27570 sys_catalog.cc:458] T 00000000000000000000000000000000 P e51ba110882f4faa9394058c43e1f5a7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:28.126008 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:28.127045 27584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:28.132793 27584 catalog_manager.cc:1347] Generated new cluster ID: db903a7a0edb4e1abcad966fbcd42a4a
I20260927 15:11:28.132901 27584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:28.139101 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:28.129282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34005 (local address 127.0.0.1:36825)
0927 15:11:28.129390 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:28.129394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:28.131408 (+ 2014us) server_negotiation.cc:407] Connection header received
0927 15:11:28.131569 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:28.131572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:28.131624 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:28.131726 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:28.132597 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.133289 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.134094 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:28.134330 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:28.134419 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:28.135825 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:28.135828 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:28.135830 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:28.135858 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:28.137305 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:28.137818 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:28.137822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:28.137823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:28.137907 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:28.138216 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:28.138220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:28.138221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:28.138410 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:28.138543 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:28.138826 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 15:11:28.138923 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:11:28.153043 27584 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:28.154242 27584 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:28.161289 27584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e51ba110882f4faa9394058c43e1f5a7: Generated new TSK 0
I20260927 15:11:28.162002 27584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:28.168009 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 27496
W20260927 15:11:28.172708 27481 connection.cc:535] client connection to 127.0.0.1:36825 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36825): Connection reset by peer (error 104)
2026-09-27T15:11:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:28 dist-test-slave-hhpq krb5kdc[27595](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:28 dist-test-slave-hhpq krb5kdc[27595](info): set up 2 sockets
Sep 27 15:11:28 dist-test-slave-hhpq krb5kdc[27595](info): commencing operation
krb5kdc: starting...
W20260927 15:11:29.120679 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:11:29 dist-test-slave-hhpq krb5kdc[27595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:11:29.241937 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46593
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:29.353971 27608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:29.354293 27608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:29.354354 27608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:29.358578 27608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:29.358657 27608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:29.358685 27608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:29.358706 27608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:29.358726 27608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:29.364347 27608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46593
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:29.365538 27608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:29.366614 27608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:29.375564 27614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:29.375589 27616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:29.375648 27608 server_base.cc:1034] running on GCE node
W20260927 15:11:29.375564 27613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:29.376365 27608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:29.377614 27608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:29.378793 27608 hybrid_clock.cc:648] HybridClock initialized: now 1790521889378775 us; error 30 us; skew 500 ppm
Sep 27 15:11:29 dist-test-slave-hhpq krb5kdc[27595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521889, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:29.381644 27608 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:11:29.382624 27608 webserver.cc:460] Webserver started at http://127.0.0.1:43463/ using document root <none> and password file <none>
I20260927 15:11:29.383201 27608 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:29.383267 27608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:29.383455 27608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:29.385588 27608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "77347321ae8f42629c3b885e1b30741b"
format_stamp: "Formatted at 2026-09-27 15:11:29 on dist-test-slave-hhpq"
server_key: "cf02535ebbe0ca46114c3c5959400250"
server_key_iv: "650e8bd6c573df3986b89f1db749d38b"
server_key_version: "encryptionkey@0"
I20260927 15:11:29.386178 27608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "77347321ae8f42629c3b885e1b30741b"
format_stamp: "Formatted at 2026-09-27 15:11:29 on dist-test-slave-hhpq"
server_key: "cf02535ebbe0ca46114c3c5959400250"
server_key_iv: "650e8bd6c573df3986b89f1db749d38b"
server_key_version: "encryptionkey@0"
I20260927 15:11:29.389897 27608 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:11:29.392336 27623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:29.393648 27608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:11:29.393769 27608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "77347321ae8f42629c3b885e1b30741b"
format_stamp: "Formatted at 2026-09-27 15:11:29 on dist-test-slave-hhpq"
server_key: "cf02535ebbe0ca46114c3c5959400250"
server_key_iv: "650e8bd6c573df3986b89f1db749d38b"
server_key_version: "encryptionkey@0"
I20260927 15:11:29.393872 27608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:29.411552 27608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:29.415156 27608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:29.415364 27608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:29.422679 27608 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46593
I20260927 15:11:29.422734 27675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46593 every 8 connection(s)
I20260927 15:11:29.423730 27608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:29.426874 27676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:29.428452 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 27608
I20260927 15:11:29.428599 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:29.428962 7239 external_mini_cluster.cc:1428] Setting key e528797491cae06c3b661673736a287a
I20260927 15:11:29.433547 27676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77347321ae8f42629c3b885e1b30741b: Bootstrap starting.
I20260927 15:11:29.436298 27676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77347321ae8f42629c3b885e1b30741b: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:29.437103 27676 log.cc:824] T 00000000000000000000000000000000 P 77347321ae8f42629c3b885e1b30741b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:29.439435 27676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77347321ae8f42629c3b885e1b30741b: No bootstrap required, opened a new log
I20260927 15:11:29.441962 27679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:29.430450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32971 (local address 127.0.0.1:46593)
0927 15:11:29.430881 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:29.430898 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:29.430936 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:11:29.431589 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:29.431607 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:29.431966 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:29.432264 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:29.432611 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:29.433689 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:29.434888 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:29.435409 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:29.435556 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:29.439752 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:29.439759 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:29.439776 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:29.440235 (+ 459us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:29.440304 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:29.440332 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:29.440464 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:29.440713 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 15:11:29.440909 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":126,"threads_started":1}
I20260927 15:11:29.442881 27676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77347321ae8f42629c3b885e1b30741b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77347321ae8f42629c3b885e1b30741b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260927 15:11:29.443187 27676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77347321ae8f42629c3b885e1b30741b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:29.443248 27676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77347321ae8f42629c3b885e1b30741b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77347321ae8f42629c3b885e1b30741b, State: Initialized, Role: FOLLOWER
I20260927 15:11:29.443727 27676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77347321ae8f42629c3b885e1b30741b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77347321ae8f42629c3b885e1b30741b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260927 15:11:29.443882 27676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77347321ae8f42629c3b885e1b30741b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:29.443948 27676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77347321ae8f42629c3b885e1b30741b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:29.444026 27676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77347321ae8f42629c3b885e1b30741b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:29.445173 27676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77347321ae8f42629c3b885e1b30741b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77347321ae8f42629c3b885e1b30741b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260927 15:11:29.445238 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:29.445564 27676 leader_election.cc:302] T 00000000000000000000000000000000 P 77347321ae8f42629c3b885e1b30741b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77347321ae8f42629c3b885e1b30741b; no voters:
I20260927 15:11:29.445935 27676 leader_election.cc:288] T 00000000000000000000000000000000 P 77347321ae8f42629c3b885e1b30741b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:29.446040 27681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77347321ae8f42629c3b885e1b30741b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:29.446301 27681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77347321ae8f42629c3b885e1b30741b [term 1 LEADER]: Becoming Leader. State: Replica: 77347321ae8f42629c3b885e1b30741b, State: Running, Role: LEADER
I20260927 15:11:29.446631 27681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77347321ae8f42629c3b885e1b30741b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77347321ae8f42629c3b885e1b30741b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260927 15:11:29.447418 27676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77347321ae8f42629c3b885e1b30741b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:29.448410 27683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77347321ae8f42629c3b885e1b30741b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77347321ae8f42629c3b885e1b30741b. Latest consensus state: current_term: 1 leader_uuid: "77347321ae8f42629c3b885e1b30741b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77347321ae8f42629c3b885e1b30741b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } } }
I20260927 15:11:29.448410 27682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77347321ae8f42629c3b885e1b30741b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77347321ae8f42629c3b885e1b30741b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77347321ae8f42629c3b885e1b30741b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } } }
I20260927 15:11:29.448566 27683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77347321ae8f42629c3b885e1b30741b [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:29.448566 27682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77347321ae8f42629c3b885e1b30741b [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:29.449033 27695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:29.452306 27695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:29.453668 27679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:29.447550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45461 (local address 127.0.0.1:46593)
0927 15:11:29.447696 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:29.447701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:29.448881 (+ 1180us) server_negotiation.cc:407] Connection header received
0927 15:11:29.449186 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:29.449190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:29.449239 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:29.449288 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:29.449712 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:29.450436 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:29.451576 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:29.451919 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:29.452020 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:29.453009 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:29.453013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:29.453016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:29.453031 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:29.453061 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:29.453076 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:29.453161 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:29.453483 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 15:11:29.453534 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 15:11:29.458431 27695 catalog_manager.cc:1347] Generated new cluster ID: 4f09f8eb719744afbd2d43cda4027ab8
I20260927 15:11:29.458506 27695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:29.463505 27695 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:29.464488 27695 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:29.470438 27695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77347321ae8f42629c3b885e1b30741b: Generated new TSK 0
I20260927 15:11:29.471306 27695 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:29.480507 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 27608
W20260927 15:11:29.485085 27593 connection.cc:535] client connection to 127.0.0.1:46593 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46593): Connection reset by peer (error 104)
2026-09-27T15:11:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:29 dist-test-slave-hhpq krb5kdc[27707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:29 dist-test-slave-hhpq krb5kdc[27707](info): set up 2 sockets
Sep 27 15:11:29 dist-test-slave-hhpq krb5kdc[27707](info): commencing operation
krb5kdc: starting...
W20260927 15:11:30.436738 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:11:30 dist-test-slave-hhpq krb5kdc[27707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:11:30.557732 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33387
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:30.666913 27720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:30.667219 27720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:30.667275 27720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:30.671370 27720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:30.671448 27720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:30.671483 27720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:30.671504 27720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:30.671523 27720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:30.677139 27720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33387
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:30.678287 27720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:30.679339 27720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:30.688073 27725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:30.688149 27720 server_base.cc:1034] running on GCE node
W20260927 15:11:30.688073 27728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:30.688091 27726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:30.689030 27720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:30.690173 27720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:30.691355 27720 hybrid_clock.cc:648] HybridClock initialized: now 1790521890691336 us; error 32 us; skew 500 ppm
Sep 27 15:11:30 dist-test-slave-hhpq krb5kdc[27707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:30.694136 27720 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:11:30.695050 27720 webserver.cc:460] Webserver started at http://127.0.0.1:35969/ using document root <none> and password file <none>
I20260927 15:11:30.695592 27720 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:30.695650 27720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:30.695823 27720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:30.697901 27720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "fd357f95d5b14f129fbf935421111c67"
format_stamp: "Formatted at 2026-09-27 15:11:30 on dist-test-slave-hhpq"
server_key: "e744d6efbb2ea29163c7e956950b231b"
server_key_iv: "5ddbe9379fbe49093854799eed0dcc5c"
server_key_version: "encryptionkey@0"
I20260927 15:11:30.698428 27720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "fd357f95d5b14f129fbf935421111c67"
format_stamp: "Formatted at 2026-09-27 15:11:30 on dist-test-slave-hhpq"
server_key: "e744d6efbb2ea29163c7e956950b231b"
server_key_iv: "5ddbe9379fbe49093854799eed0dcc5c"
server_key_version: "encryptionkey@0"
I20260927 15:11:30.702126 27720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:11:30.704662 27735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:30.706025 27720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:30.706132 27720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "fd357f95d5b14f129fbf935421111c67"
format_stamp: "Formatted at 2026-09-27 15:11:30 on dist-test-slave-hhpq"
server_key: "e744d6efbb2ea29163c7e956950b231b"
server_key_iv: "5ddbe9379fbe49093854799eed0dcc5c"
server_key_version: "encryptionkey@0"
I20260927 15:11:30.706257 27720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:30.711091 27720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:30.714466 27720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:30.714645 27720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:30.721582 27720 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33387
I20260927 15:11:30.721640 27787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33387 every 8 connection(s)
I20260927 15:11:30.722555 27720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:30.723795 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 27720
I20260927 15:11:30.723943 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:30.724263 7239 external_mini_cluster.cc:1428] Setting key cd6efcc5910488bb49edc37cbf210931
I20260927 15:11:30.726051 27788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:30.729570 27791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:30.725661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50513 (local address 127.0.0.1:33387)
0927 15:11:30.726085 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:30.726102 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:30.726148 (+ 46us) server_negotiation.cc:407] Connection header received
0927 15:11:30.726816 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:30.726835 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:30.727152 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:30.727451 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:30.728315 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:30.728320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:30.728335 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:30.728345 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:30.728399 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:30.728418 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:30.728552 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:30.728612 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 15:11:30.728757 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":128,"threads_started":1}
I20260927 15:11:30.732642 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:30.732774 27788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd357f95d5b14f129fbf935421111c67: Bootstrap starting.
I20260927 15:11:30.735591 27788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd357f95d5b14f129fbf935421111c67: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:30.736418 27788 log.cc:824] T 00000000000000000000000000000000 P fd357f95d5b14f129fbf935421111c67: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:30.736662 27791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:30.734981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54031 (local address 127.0.0.1:33387)
0927 15:11:30.735106 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:30.735111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:30.735323 (+ 212us) server_negotiation.cc:407] Connection header received
0927 15:11:30.735426 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:30.735429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:30.735469 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:30.735560 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:30.736311 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:30.736314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:30.736317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:30.736328 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:30.736358 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:30.736370 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:30.736471 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:30.736512 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 15:11:30.736551 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:11:30.738579 27788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd357f95d5b14f129fbf935421111c67: No bootstrap required, opened a new log
I20260927 15:11:30.742225 27788 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd357f95d5b14f129fbf935421111c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd357f95d5b14f129fbf935421111c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33387 } }
I20260927 15:11:30.742528 27788 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd357f95d5b14f129fbf935421111c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:30.742605 27788 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd357f95d5b14f129fbf935421111c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd357f95d5b14f129fbf935421111c67, State: Initialized, Role: FOLLOWER
I20260927 15:11:30.743059 27788 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd357f95d5b14f129fbf935421111c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd357f95d5b14f129fbf935421111c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33387 } }
I20260927 15:11:30.743191 27788 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd357f95d5b14f129fbf935421111c67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:30.743238 27788 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd357f95d5b14f129fbf935421111c67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:30.743340 27788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd357f95d5b14f129fbf935421111c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:30.744297 27788 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd357f95d5b14f129fbf935421111c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd357f95d5b14f129fbf935421111c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33387 } }
I20260927 15:11:30.744644 27788 leader_election.cc:302] T 00000000000000000000000000000000 P fd357f95d5b14f129fbf935421111c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd357f95d5b14f129fbf935421111c67; no voters:
I20260927 15:11:30.745031 27788 leader_election.cc:288] T 00000000000000000000000000000000 P fd357f95d5b14f129fbf935421111c67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:30.745134 27798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd357f95d5b14f129fbf935421111c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:30.745344 27798 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd357f95d5b14f129fbf935421111c67 [term 1 LEADER]: Becoming Leader. State: Replica: fd357f95d5b14f129fbf935421111c67, State: Running, Role: LEADER
I20260927 15:11:30.745692 27798 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd357f95d5b14f129fbf935421111c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd357f95d5b14f129fbf935421111c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33387 } }
I20260927 15:11:30.746287 27788 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd357f95d5b14f129fbf935421111c67 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:30.747386 27799 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd357f95d5b14f129fbf935421111c67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd357f95d5b14f129fbf935421111c67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd357f95d5b14f129fbf935421111c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33387 } } }
I20260927 15:11:30.747526 27799 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd357f95d5b14f129fbf935421111c67 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:30.747924 27807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:30.747887 27800 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd357f95d5b14f129fbf935421111c67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd357f95d5b14f129fbf935421111c67. Latest consensus state: current_term: 1 leader_uuid: "fd357f95d5b14f129fbf935421111c67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd357f95d5b14f129fbf935421111c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33387 } } }
I20260927 15:11:30.748035 27800 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd357f95d5b14f129fbf935421111c67 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:30.751344 27807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:30.757436 27807 catalog_manager.cc:1347] Generated new cluster ID: c4a467f8b5fc4d1d86b5bb97478f3649
I20260927 15:11:30.757521 27807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:30.766009 27807 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:30.766400 27807 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:30.773640 27807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd357f95d5b14f129fbf935421111c67: Generated new TSK 0
I20260927 15:11:30.774525 27807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:30.781009 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 27720
2026-09-27T15:11:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:30 dist-test-slave-hhpq krb5kdc[27819](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:30 dist-test-slave-hhpq krb5kdc[27819](info): set up 2 sockets
Sep 27 15:11:30 dist-test-slave-hhpq krb5kdc[27819](info): commencing operation
krb5kdc: starting...
W20260927 15:11:31.732893 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:11:31 dist-test-slave-hhpq krb5kdc[27819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:11:31.852226 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44701
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:31.964470 27832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:31.964793 27832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:31.964881 27832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:31.969116 27832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:31.969197 27832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:31.969224 27832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:31.969244 27832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:31.969264 27832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:31.974874 27832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:31.976115 27832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:31.977201 27832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:31.985735 27838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:31.985735 27837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:31.985755 27840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:31.986061 27832 server_base.cc:1034] running on GCE node
I20260927 15:11:31.986758 27832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:31.987926 27832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:31.989085 27832 hybrid_clock.cc:648] HybridClock initialized: now 1790521891989068 us; error 35 us; skew 500 ppm
Sep 27 15:11:31 dist-test-slave-hhpq krb5kdc[27819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:31.997499 27832 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:11:31.998557 27832 webserver.cc:460] Webserver started at http://127.0.0.1:43637/ using document root <none> and password file <none>
I20260927 15:11:31.999118 27832 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:31.999177 27832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:31.999353 27832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:32.001646 27832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "388401b362ee4ed3811b9b17114de395"
format_stamp: "Formatted at 2026-09-27 15:11:31 on dist-test-slave-hhpq"
server_key: "6fbc9f7f39fe75ad8c83493c7c91a9a5"
server_key_iv: "fd98d8bd2cb167e2b6cd62fee2d54e80"
server_key_version: "encryptionkey@0"
I20260927 15:11:32.002204 27832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "388401b362ee4ed3811b9b17114de395"
format_stamp: "Formatted at 2026-09-27 15:11:31 on dist-test-slave-hhpq"
server_key: "6fbc9f7f39fe75ad8c83493c7c91a9a5"
server_key_iv: "fd98d8bd2cb167e2b6cd62fee2d54e80"
server_key_version: "encryptionkey@0"
I20260927 15:11:32.006224 27832 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 15:11:32.009114 27847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:32.010520 27832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:11:32.010692 27832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "388401b362ee4ed3811b9b17114de395"
format_stamp: "Formatted at 2026-09-27 15:11:31 on dist-test-slave-hhpq"
server_key: "6fbc9f7f39fe75ad8c83493c7c91a9a5"
server_key_iv: "fd98d8bd2cb167e2b6cd62fee2d54e80"
server_key_version: "encryptionkey@0"
I20260927 15:11:32.010818 27832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:32.029343 27832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:32.032727 27832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:32.032954 27832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:32.040400 27832 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44701
I20260927 15:11:32.040421 27899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44701 every 8 connection(s)
I20260927 15:11:32.041436 27832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:32.044273 27900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:32.049021 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 27832
I20260927 15:11:32.049190 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:32.049443 7239 external_mini_cluster.cc:1428] Setting key 4596b55513d45f87a6a9631656bb838f
I20260927 15:11:32.050624 27900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 388401b362ee4ed3811b9b17114de395: Bootstrap starting.
I20260927 15:11:32.053344 27900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 388401b362ee4ed3811b9b17114de395: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:32.054412 27900 log.cc:824] T 00000000000000000000000000000000 P 388401b362ee4ed3811b9b17114de395: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:32.057206 27900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 388401b362ee4ed3811b9b17114de395: No bootstrap required, opened a new log
Sep 27 15:11:32 dist-test-slave-hhpq krb5kdc[27819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:11:32.061547 27900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 388401b362ee4ed3811b9b17114de395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388401b362ee4ed3811b9b17114de395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44701 } }
I20260927 15:11:32.061815 27900 raft_consensus.cc:374] T 00000000000000000000000000000000 P 388401b362ee4ed3811b9b17114de395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:32.061883 27900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 388401b362ee4ed3811b9b17114de395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 388401b362ee4ed3811b9b17114de395, State: Initialized, Role: FOLLOWER
I20260927 15:11:32.062445 27900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 388401b362ee4ed3811b9b17114de395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388401b362ee4ed3811b9b17114de395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44701 } }
I20260927 15:11:32.062600 27900 raft_consensus.cc:388] T 00000000000000000000000000000000 P 388401b362ee4ed3811b9b17114de395 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:32.062671 27900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 388401b362ee4ed3811b9b17114de395 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:32.062775 27900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 388401b362ee4ed3811b9b17114de395 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:32.063743 27900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 388401b362ee4ed3811b9b17114de395 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388401b362ee4ed3811b9b17114de395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44701 } }
I20260927 15:11:32.064111 27900 leader_election.cc:302] T 00000000000000000000000000000000 P 388401b362ee4ed3811b9b17114de395 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 388401b362ee4ed3811b9b17114de395; no voters:
I20260927 15:11:32.064452 27900 leader_election.cc:288] T 00000000000000000000000000000000 P 388401b362ee4ed3811b9b17114de395 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:32.064538 27905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 388401b362ee4ed3811b9b17114de395 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:32.064978 27905 raft_consensus.cc:686] T 00000000000000000000000000000000 P 388401b362ee4ed3811b9b17114de395 [term 1 LEADER]: Becoming Leader. State: Replica: 388401b362ee4ed3811b9b17114de395, State: Running, Role: LEADER
I20260927 15:11:32.065325 27905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 388401b362ee4ed3811b9b17114de395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388401b362ee4ed3811b9b17114de395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44701 } }
I20260927 15:11:32.065806 27903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:32.051073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41883 (local address 127.0.0.1:44701)
0927 15:11:32.051541 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:32.051558 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:32.051598 (+ 40us) server_negotiation.cc:407] Connection header received
0927 15:11:32.052319 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:32.052340 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:32.052750 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:32.053140 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:32.054241 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.055786 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.057174 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.057749 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.057896 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:32.060752 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:32.060758 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:32.060775 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:32.060809 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:32.062667 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:32.063177 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:32.063184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:32.063191 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:32.063318 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:32.063583 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:32.063586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:32.063588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:32.064058 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:32.064270 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:32.064553 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 15:11:32.064916 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":131,"threads_started":1}
I20260927 15:11:32.066012 27900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 388401b362ee4ed3811b9b17114de395 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:32.066744 27905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 388401b362ee4ed3811b9b17114de395 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 388401b362ee4ed3811b9b17114de395. Latest consensus state: current_term: 1 leader_uuid: "388401b362ee4ed3811b9b17114de395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388401b362ee4ed3811b9b17114de395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44701 } } }
I20260927 15:11:32.067906 27905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 388401b362ee4ed3811b9b17114de395 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:32.068378 27916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:32.069401 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:32.070605 27906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 388401b362ee4ed3811b9b17114de395 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "388401b362ee4ed3811b9b17114de395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388401b362ee4ed3811b9b17114de395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44701 } } }
I20260927 15:11:32.070727 27906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 388401b362ee4ed3811b9b17114de395 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:32.072525 27916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:32.079144 27916 catalog_manager.cc:1347] Generated new cluster ID: 59f484abbd874fa3a4d1163224702237
I20260927 15:11:32.079216 27916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:32.082834 27903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:32.073327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60213 (local address 127.0.0.1:44701)
0927 15:11:32.073440 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:32.073444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:32.074925 (+ 1481us) server_negotiation.cc:407] Connection header received
0927 15:11:32.075073 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:32.075076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:32.075132 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:32.075222 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:32.076078 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.076760 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.077681 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:32.077913 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:32.077997 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:32.079347 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:32.079350 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:32.079353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:32.079382 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:32.081123 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:32.081597 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:32.081601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:32.081603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:32.081703 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:32.081961 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:32.081965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:32.081967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:32.082203 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:32.082330 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:32.082536 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 15:11:32.082665 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:11:32.092291 27916 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:32.093367 27916 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:32.101183 27916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 388401b362ee4ed3811b9b17114de395: Generated new TSK 0
I20260927 15:11:32.102149 27916 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:32.111524 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 27832
2026-09-27T15:11:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:32 dist-test-slave-hhpq krb5kdc[27931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:32 dist-test-slave-hhpq krb5kdc[27931](info): set up 2 sockets
Sep 27 15:11:32 dist-test-slave-hhpq krb5kdc[27931](info): commencing operation
krb5kdc: starting...
W20260927 15:11:33.069006 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:11:33 dist-test-slave-hhpq krb5kdc[27931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:11:33.192476 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:33.304126 27944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:33.304453 27944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:33.304500 27944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:33.308687 27944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:33.308766 27944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:33.308786 27944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:33.308802 27944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:33.308816 27944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:33.314697 27944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:33.315876 27944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:33.317054 27944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:33.325986 27950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:33.326046 27949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:33.326078 27944 server_base.cc:1034] running on GCE node
W20260927 15:11:33.326011 27952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:33.327006 27944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:33.328197 27944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:33.329363 27944 hybrid_clock.cc:648] HybridClock initialized: now 1790521893329346 us; error 40 us; skew 500 ppm
Sep 27 15:11:33 dist-test-slave-hhpq krb5kdc[27931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:33.332144 27944 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:11:33.333248 27944 webserver.cc:460] Webserver started at http://127.0.0.1:36981/ using document root <none> and password file <none>
I20260927 15:11:33.333829 27944 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:33.333899 27944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:33.334105 27944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:33.336205 27944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "1674c73ffadd4c8f80ae585d085e0b81"
format_stamp: "Formatted at 2026-09-27 15:11:33 on dist-test-slave-hhpq"
server_key: "1c4af899e3eec7f0bc211786f62b2cb3"
server_key_iv: "1d00c158680575bcd85d7a7fbad7c1e8"
server_key_version: "encryptionkey@0"
I20260927 15:11:33.336774 27944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "1674c73ffadd4c8f80ae585d085e0b81"
format_stamp: "Formatted at 2026-09-27 15:11:33 on dist-test-slave-hhpq"
server_key: "1c4af899e3eec7f0bc211786f62b2cb3"
server_key_iv: "1d00c158680575bcd85d7a7fbad7c1e8"
server_key_version: "encryptionkey@0"
I20260927 15:11:33.340746 27944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:11:33.343369 27959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:33.344960 27944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:33.345103 27944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "1674c73ffadd4c8f80ae585d085e0b81"
format_stamp: "Formatted at 2026-09-27 15:11:33 on dist-test-slave-hhpq"
server_key: "1c4af899e3eec7f0bc211786f62b2cb3"
server_key_iv: "1d00c158680575bcd85d7a7fbad7c1e8"
server_key_version: "encryptionkey@0"
I20260927 15:11:33.345222 27944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:33.370657 27944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:33.373939 27944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:33.374145 27944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:33.381405 27944 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32889
I20260927 15:11:33.381419 28011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32889 every 8 connection(s)
I20260927 15:11:33.382419 27944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:33.385391 28012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:33.389387 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 27944
I20260927 15:11:33.389539 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:33.389793 7239 external_mini_cluster.cc:1428] Setting key 3660d2b3c9c4edda960b3dacdc010699
I20260927 15:11:33.391623 28012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1674c73ffadd4c8f80ae585d085e0b81: Bootstrap starting.
I20260927 15:11:33.394150 28012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1674c73ffadd4c8f80ae585d085e0b81: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:33.394958 28012 log.cc:824] T 00000000000000000000000000000000 P 1674c73ffadd4c8f80ae585d085e0b81: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:33.397346 28012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1674c73ffadd4c8f80ae585d085e0b81: No bootstrap required, opened a new log
I20260927 15:11:33.399636 28015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:33.391434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41409 (local address 127.0.0.1:32889)
0927 15:11:33.391851 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:33.391866 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:33.391904 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:11:33.392602 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:33.392621 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:33.393056 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:33.393360 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:33.393709 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:33.394775 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:33.395896 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:33.396456 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:33.396602 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:33.397499 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:33.397505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:33.397520 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:33.398009 (+ 489us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:33.398071 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:33.398097 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:33.398179 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:33.398383 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 15:11:33.398621 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":108,"threads_started":1}
I20260927 15:11:33.400609 28012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1674c73ffadd4c8f80ae585d085e0b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1674c73ffadd4c8f80ae585d085e0b81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32889 } }
I20260927 15:11:33.400872 28012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1674c73ffadd4c8f80ae585d085e0b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:33.400938 28012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1674c73ffadd4c8f80ae585d085e0b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1674c73ffadd4c8f80ae585d085e0b81, State: Initialized, Role: FOLLOWER
I20260927 15:11:33.401364 28012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1674c73ffadd4c8f80ae585d085e0b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1674c73ffadd4c8f80ae585d085e0b81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32889 } }
I20260927 15:11:33.401501 28012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1674c73ffadd4c8f80ae585d085e0b81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:33.401575 28012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1674c73ffadd4c8f80ae585d085e0b81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:33.401684 28012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1674c73ffadd4c8f80ae585d085e0b81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:33.402673 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:33.402704 28012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1674c73ffadd4c8f80ae585d085e0b81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1674c73ffadd4c8f80ae585d085e0b81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32889 } }
I20260927 15:11:33.403074 28012 leader_election.cc:302] T 00000000000000000000000000000000 P 1674c73ffadd4c8f80ae585d085e0b81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1674c73ffadd4c8f80ae585d085e0b81; no voters:
I20260927 15:11:33.403453 28012 leader_election.cc:288] T 00000000000000000000000000000000 P 1674c73ffadd4c8f80ae585d085e0b81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:33.403548 28017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1674c73ffadd4c8f80ae585d085e0b81 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:33.403764 28017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1674c73ffadd4c8f80ae585d085e0b81 [term 1 LEADER]: Becoming Leader. State: Replica: 1674c73ffadd4c8f80ae585d085e0b81, State: Running, Role: LEADER
I20260927 15:11:33.404148 28017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1674c73ffadd4c8f80ae585d085e0b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1674c73ffadd4c8f80ae585d085e0b81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32889 } }
I20260927 15:11:33.404886 28012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1674c73ffadd4c8f80ae585d085e0b81 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:33.405692 28018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1674c73ffadd4c8f80ae585d085e0b81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1674c73ffadd4c8f80ae585d085e0b81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1674c73ffadd4c8f80ae585d085e0b81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32889 } } }
I20260927 15:11:33.405825 28018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1674c73ffadd4c8f80ae585d085e0b81 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:33.406201 28031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:33.406201 28019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1674c73ffadd4c8f80ae585d085e0b81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1674c73ffadd4c8f80ae585d085e0b81. Latest consensus state: current_term: 1 leader_uuid: "1674c73ffadd4c8f80ae585d085e0b81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1674c73ffadd4c8f80ae585d085e0b81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32889 } } }
I20260927 15:11:33.406298 28019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1674c73ffadd4c8f80ae585d085e0b81 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:33.409693 28031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:33.410883 28015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:33.405245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59355 (local address 127.0.0.1:32889)
0927 15:11:33.405350 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:33.405355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:33.406589 (+ 1234us) server_negotiation.cc:407] Connection header received
0927 15:11:33.406719 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:33.406723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:33.406776 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:33.406880 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:33.407334 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:33.408062 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:33.409166 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:33.409526 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:33.409657 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:33.410489 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:33.410493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:33.410496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:33.410510 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:33.410549 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:33.410563 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:33.410625 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:33.410702 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 15:11:33.410754 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:11:33.416464 28031 catalog_manager.cc:1347] Generated new cluster ID: 67f4c3deb6464fd295ce6eadd899dfc5
I20260927 15:11:33.416559 28031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:33.445179 28031 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:33.446169 28031 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:33.450536 28031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1674c73ffadd4c8f80ae585d085e0b81: Generated new TSK 0
I20260927 15:11:33.451391 28031 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:33.461750 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 27944
2026-09-27T15:11:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:33 dist-test-slave-hhpq krb5kdc[28043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:33 dist-test-slave-hhpq krb5kdc[28043](info): set up 2 sockets
Sep 27 15:11:33 dist-test-slave-hhpq krb5kdc[28043](info): commencing operation
krb5kdc: starting...
W20260927 15:11:34.429095 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:11:34 dist-test-slave-hhpq krb5kdc[28043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:11:34.548388 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:34.659855 28056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:34.660179 28056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:34.660243 28056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:34.664448 28056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:34.664536 28056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:34.664564 28056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:34.664585 28056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:34.664608 28056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:34.670226 28056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:34.671379 28056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:34.672447 28056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:34.681442 28062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:34.681452 28061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:34.681448 28064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:34.681705 28056 server_base.cc:1034] running on GCE node
I20260927 15:11:34.682319 28056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:34.683539 28056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:34.684762 28056 hybrid_clock.cc:648] HybridClock initialized: now 1790521894684728 us; error 54 us; skew 500 ppm
Sep 27 15:11:34 dist-test-slave-hhpq krb5kdc[28043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:34.688169 28056 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:11:34.689312 28056 webserver.cc:460] Webserver started at http://127.0.0.1:45871/ using document root <none> and password file <none>
I20260927 15:11:34.689931 28056 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:34.690002 28056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:34.690203 28056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:34.692396 28056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "02aff645d0754aa6963630712b0c2df9"
format_stamp: "Formatted at 2026-09-27 15:11:34 on dist-test-slave-hhpq"
server_key: "ff0ac06f16d61612f20bcf6bdea5fc1b"
server_key_iv: "ee791b55c0608a681ec50aeb2bf21140"
server_key_version: "encryptionkey@0"
I20260927 15:11:34.693034 28056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "02aff645d0754aa6963630712b0c2df9"
format_stamp: "Formatted at 2026-09-27 15:11:34 on dist-test-slave-hhpq"
server_key: "ff0ac06f16d61612f20bcf6bdea5fc1b"
server_key_iv: "ee791b55c0608a681ec50aeb2bf21140"
server_key_version: "encryptionkey@0"
I20260927 15:11:34.697158 28056 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:11:34.699990 28071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:34.701498 28056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:34.701632 28056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "02aff645d0754aa6963630712b0c2df9"
format_stamp: "Formatted at 2026-09-27 15:11:34 on dist-test-slave-hhpq"
server_key: "ff0ac06f16d61612f20bcf6bdea5fc1b"
server_key_iv: "ee791b55c0608a681ec50aeb2bf21140"
server_key_version: "encryptionkey@0"
I20260927 15:11:34.701751 28056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:34.706478 28056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:34.709631 28056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:34.709802 28056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:34.717175 28056 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35325
I20260927 15:11:34.717233 28123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35325 every 8 connection(s)
I20260927 15:11:34.718204 28056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:34.721261 28124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:34.724751 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 28056
I20260927 15:11:34.724932 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:34.725237 7239 external_mini_cluster.cc:1428] Setting key d520ea453cfc3c38d821e541f48fd631
I20260927 15:11:34.727723 28124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02aff645d0754aa6963630712b0c2df9: Bootstrap starting.
I20260927 15:11:34.730268 28124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02aff645d0754aa6963630712b0c2df9: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:34.730646 28127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:34.726696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53641 (local address 127.0.0.1:35325)
0927 15:11:34.727155 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:34.727172 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:34.727207 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:11:34.727855 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:34.727879 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:34.728206 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:34.728499 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:34.729356 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:34.729360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:34.729373 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:34.729383 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:34.729444 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:34.729465 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:34.729572 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:34.729664 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 15:11:34.729814 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":147,"threads_started":1}
I20260927 15:11:34.731105 28124 log.cc:824] T 00000000000000000000000000000000 P 02aff645d0754aa6963630712b0c2df9: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:34.733273 28124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02aff645d0754aa6963630712b0c2df9: No bootstrap required, opened a new log
I20260927 15:11:34.733537 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:34.737198 28127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:34.735486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54393 (local address 127.0.0.1:35325)
0927 15:11:34.735607 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:34.735611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:34.735728 (+ 117us) server_negotiation.cc:407] Connection header received
0927 15:11:34.735823 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:34.735826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:34.735867 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:34.735949 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:34.736644 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:34.736648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:34.736650 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:34.736663 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:34.736691 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:34.736704 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:34.736790 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:34.737038 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 15:11:34.737078 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:11:34.737447 28124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02aff645d0754aa6963630712b0c2df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02aff645d0754aa6963630712b0c2df9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } }
I20260927 15:11:34.737703 28124 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02aff645d0754aa6963630712b0c2df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:34.737751 28124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02aff645d0754aa6963630712b0c2df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02aff645d0754aa6963630712b0c2df9, State: Initialized, Role: FOLLOWER
I20260927 15:11:34.738189 28124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02aff645d0754aa6963630712b0c2df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02aff645d0754aa6963630712b0c2df9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } }
I20260927 15:11:34.738307 28124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02aff645d0754aa6963630712b0c2df9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:34.738375 28124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02aff645d0754aa6963630712b0c2df9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:34.738471 28124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02aff645d0754aa6963630712b0c2df9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:34.739442 28124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02aff645d0754aa6963630712b0c2df9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02aff645d0754aa6963630712b0c2df9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } }
I20260927 15:11:34.739795 28124 leader_election.cc:302] T 00000000000000000000000000000000 P 02aff645d0754aa6963630712b0c2df9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02aff645d0754aa6963630712b0c2df9; no voters:
I20260927 15:11:34.740109 28124 leader_election.cc:288] T 00000000000000000000000000000000 P 02aff645d0754aa6963630712b0c2df9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:34.740257 28134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02aff645d0754aa6963630712b0c2df9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:34.740564 28134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02aff645d0754aa6963630712b0c2df9 [term 1 LEADER]: Becoming Leader. State: Replica: 02aff645d0754aa6963630712b0c2df9, State: Running, Role: LEADER
I20260927 15:11:34.740901 28134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02aff645d0754aa6963630712b0c2df9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02aff645d0754aa6963630712b0c2df9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } }
I20260927 15:11:34.741400 28124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02aff645d0754aa6963630712b0c2df9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:34.742568 28135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02aff645d0754aa6963630712b0c2df9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02aff645d0754aa6963630712b0c2df9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02aff645d0754aa6963630712b0c2df9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } } }
I20260927 15:11:34.742689 28135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02aff645d0754aa6963630712b0c2df9 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:34.743089 28136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02aff645d0754aa6963630712b0c2df9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02aff645d0754aa6963630712b0c2df9. Latest consensus state: current_term: 1 leader_uuid: "02aff645d0754aa6963630712b0c2df9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02aff645d0754aa6963630712b0c2df9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } } }
I20260927 15:11:34.743237 28136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02aff645d0754aa6963630712b0c2df9 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:34.744222 28143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:34.747124 28143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:34.752547 28143 catalog_manager.cc:1347] Generated new cluster ID: 6a1656cf24d7465cb12af756096ebc1d
I20260927 15:11:34.752620 28143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:34.761150 28143 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:34.761505 28143 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:34.765892 28143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02aff645d0754aa6963630712b0c2df9: Generated new TSK 0
I20260927 15:11:34.766661 28143 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:34.775238 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 28056
2026-09-27T15:11:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:34 dist-test-slave-hhpq krb5kdc[28155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:34 dist-test-slave-hhpq krb5kdc[28155](info): set up 2 sockets
Sep 27 15:11:34 dist-test-slave-hhpq krb5kdc[28155](info): commencing operation
krb5kdc: starting...
W20260927 15:11:35.744804 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:11:35 dist-test-slave-hhpq krb5kdc[28155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:11:35.865185 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37963
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:35.978789 28168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:35.979094 28168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:35.979136 28168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:35.983368 28168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:35.983449 28168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:35.983469 28168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:35.983484 28168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:35.983497 28168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:35.989143 28168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:35.990257 28168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:35.991346 28168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:36.000285 28174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:36.000300 28173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:36.000300 28176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:36.000355 28168 server_base.cc:1034] running on GCE node
I20260927 15:11:36.001242 28168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:36.002435 28168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:36.003633 28168 hybrid_clock.cc:648] HybridClock initialized: now 1790521896003612 us; error 37 us; skew 500 ppm
Sep 27 15:11:36 dist-test-slave-hhpq krb5kdc[28155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:36.006634 28168 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:11:36.007642 28168 webserver.cc:460] Webserver started at http://127.0.0.1:39529/ using document root <none> and password file <none>
I20260927 15:11:36.008273 28168 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:36.008343 28168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:36.008545 28168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:36.010689 28168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "b6847cd6ba0d41ddae2279eb95a7e19b"
format_stamp: "Formatted at 2026-09-27 15:11:36 on dist-test-slave-hhpq"
server_key: "1c7434222cc1ca463015cac4162afb87"
server_key_iv: "0179e93210afbee7ba2ba0bb5263d69d"
server_key_version: "encryptionkey@0"
I20260927 15:11:36.011262 28168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "b6847cd6ba0d41ddae2279eb95a7e19b"
format_stamp: "Formatted at 2026-09-27 15:11:36 on dist-test-slave-hhpq"
server_key: "1c7434222cc1ca463015cac4162afb87"
server_key_iv: "0179e93210afbee7ba2ba0bb5263d69d"
server_key_version: "encryptionkey@0"
I20260927 15:11:36.015304 28168 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 15:11:36.018057 28183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:36.019469 28168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:36.019614 28168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "b6847cd6ba0d41ddae2279eb95a7e19b"
format_stamp: "Formatted at 2026-09-27 15:11:36 on dist-test-slave-hhpq"
server_key: "1c7434222cc1ca463015cac4162afb87"
server_key_iv: "0179e93210afbee7ba2ba0bb5263d69d"
server_key_version: "encryptionkey@0"
I20260927 15:11:36.019739 28168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:36.031739 28168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:36.035194 28168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:36.035396 28168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:36.042722 28168 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37963
I20260927 15:11:36.042783 28235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37963 every 8 connection(s)
I20260927 15:11:36.043749 28168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:36.046818 28236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:36.051712 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 28168
I20260927 15:11:36.051862 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:36.052155 7239 external_mini_cluster.cc:1428] Setting key 365e1e0806ebe06c1a3fe0ee3c00d1ad
I20260927 15:11:36.052760 28236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b: Bootstrap starting.
I20260927 15:11:36.055495 28236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:36.056270 28236 log.cc:824] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:36.058264 28236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b: No bootstrap required, opened a new log
Sep 27 15:11:36 dist-test-slave-hhpq krb5kdc[28155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:11:36.062263 28236 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6847cd6ba0d41ddae2279eb95a7e19b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37963 } }
I20260927 15:11:36.062520 28236 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:36.062569 28236 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6847cd6ba0d41ddae2279eb95a7e19b, State: Initialized, Role: FOLLOWER
I20260927 15:11:36.063074 28236 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6847cd6ba0d41ddae2279eb95a7e19b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37963 } }
I20260927 15:11:36.063215 28236 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:36.063313 28236 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:36.063427 28236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:36.064374 28236 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6847cd6ba0d41ddae2279eb95a7e19b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37963 } }
I20260927 15:11:36.064754 28236 leader_election.cc:302] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6847cd6ba0d41ddae2279eb95a7e19b; no voters:
I20260927 15:11:36.065167 28236 leader_election.cc:288] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:36.065263 28241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:36.065500 28241 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b [term 1 LEADER]: Becoming Leader. State: Replica: b6847cd6ba0d41ddae2279eb95a7e19b, State: Running, Role: LEADER
I20260927 15:11:36.065910 28241 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6847cd6ba0d41ddae2279eb95a7e19b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37963 } }
I20260927 15:11:36.066592 28236 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:36.067998 28239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:36.053898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:38853 (local address 127.0.0.1:37963)
0927 15:11:36.054413 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:36.054428 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:36.054467 (+ 39us) server_negotiation.cc:407] Connection header received
0927 15:11:36.055135 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:36.055155 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:36.055554 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:36.055907 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:36.056719 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.057763 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.058669 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.059171 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.059319 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:36.062134 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:36.062140 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:36.062156 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:36.062189 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:36.064034 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:36.064591 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:36.064597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:36.064603 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:36.064732 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:36.065104 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:36.065108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:36.065109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:36.065711 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:36.065943 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:36.066625 (+ 682us) server_negotiation.cc:299] Negotiation successful
0927 15:11:36.066886 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":164,"threads_started":1}
I20260927 15:11:36.069525 28242 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6847cd6ba0d41ddae2279eb95a7e19b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6847cd6ba0d41ddae2279eb95a7e19b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37963 } } }
I20260927 15:11:36.069594 28242 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:36.069342 28243 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6847cd6ba0d41ddae2279eb95a7e19b. Latest consensus state: current_term: 1 leader_uuid: "b6847cd6ba0d41ddae2279eb95a7e19b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6847cd6ba0d41ddae2279eb95a7e19b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37963 } } }
I20260927 15:11:36.069824 28243 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:36.071581 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 15:11:36.072109 28256 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:11:36.072192 28256 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:11:36.072276 28257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:36.073167 28257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:36.079172 28257 catalog_manager.cc:1347] Generated new cluster ID: 674d61a835fd47bfa48ce1ef4119a39a
I20260927 15:11:36.079241 28257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:36.085119 28257 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:36.085276 28239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:36.075576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:48735 (local address 127.0.0.1:37963)
0927 15:11:36.075687 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:36.075691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:36.075982 (+ 291us) server_negotiation.cc:407] Connection header received
0927 15:11:36.076099 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:36.076102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:36.076152 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:36.076243 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:36.077143 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.077751 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.078908 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:36.079306 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:36.079383 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:36.081325 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:36.081329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:36.081333 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:36.081361 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:36.083135 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:36.083726 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:36.083730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:36.083732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:36.083817 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:36.084332 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:36.084336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:36.084337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:36.084568 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:36.084707 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:36.085035 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 15:11:36.085126 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 15:11:36.086503 28257 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:36.102840 28257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6847cd6ba0d41ddae2279eb95a7e19b: Generated new TSK 0
I20260927 15:11:36.103813 28257 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:36.109916 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 28168
W20260927 15:11:36.114580 28153 connection.cc:535] client connection to 127.0.0.1:37963 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37963): Connection reset by peer (error 104)
2026-09-27T15:11:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:36 dist-test-slave-hhpq krb5kdc[28267](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:36 dist-test-slave-hhpq krb5kdc[28267](info): set up 2 sockets
Sep 27 15:11:36 dist-test-slave-hhpq krb5kdc[28267](info): commencing operation
krb5kdc: starting...
W20260927 15:11:37.068904 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:11:37 dist-test-slave-hhpq krb5kdc[28267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:11:37.189284 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35159
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:37.299968 28280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:37.300285 28280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:37.300348 28280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:37.304702 28280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:37.304778 28280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:37.304804 28280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:37.304845 28280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:37.304868 28280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:37.310539 28280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:37.311709 28280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:37.312767 28280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:37.321528 28286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:37.321539 28288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:37.321597 28280 server_base.cc:1034] running on GCE node
W20260927 15:11:37.321612 28285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:37.322484 28280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:37.323701 28280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:37.324941 28280 hybrid_clock.cc:648] HybridClock initialized: now 1790521897324917 us; error 40 us; skew 500 ppm
Sep 27 15:11:37 dist-test-slave-hhpq krb5kdc[28267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:37.331714 28280 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:11:37.332864 28280 webserver.cc:460] Webserver started at http://127.0.0.1:33667/ using document root <none> and password file <none>
I20260927 15:11:37.333473 28280 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:37.333545 28280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:37.333750 28280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:37.335976 28280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "1446393417e0492ea11839200eac337f"
format_stamp: "Formatted at 2026-09-27 15:11:37 on dist-test-slave-hhpq"
server_key: "5bbdd9f71e522dd6f9a265ec0f3c2cd8"
server_key_iv: "67f1722f3b3505c22e8bb096bfa08f22"
server_key_version: "encryptionkey@0"
I20260927 15:11:37.336552 28280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "1446393417e0492ea11839200eac337f"
format_stamp: "Formatted at 2026-09-27 15:11:37 on dist-test-slave-hhpq"
server_key: "5bbdd9f71e522dd6f9a265ec0f3c2cd8"
server_key_iv: "67f1722f3b3505c22e8bb096bfa08f22"
server_key_version: "encryptionkey@0"
I20260927 15:11:37.340567 28280 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 15:11:37.343178 28295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:37.344496 28280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 15:11:37.344617 28280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "1446393417e0492ea11839200eac337f"
format_stamp: "Formatted at 2026-09-27 15:11:37 on dist-test-slave-hhpq"
server_key: "5bbdd9f71e522dd6f9a265ec0f3c2cd8"
server_key_iv: "67f1722f3b3505c22e8bb096bfa08f22"
server_key_version: "encryptionkey@0"
I20260927 15:11:37.344725 28280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:37.367733 28280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:37.371057 28280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:37.371246 28280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:37.378185 28280 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35159
I20260927 15:11:37.378206 28347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35159 every 8 connection(s)
I20260927 15:11:37.379179 28280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:37.382107 28348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:37.385906 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 28280
I20260927 15:11:37.386055 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:37.386349 7239 external_mini_cluster.cc:1428] Setting key 7197f3dd347807fcd3884fc6251606f2
I20260927 15:11:37.388567 28348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f: Bootstrap starting.
I20260927 15:11:37.391168 28348 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:37.392145 28348 log.cc:824] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:37.394690 28348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f: No bootstrap required, opened a new log
I20260927 15:11:37.396440 28351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:37.387855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:39021 (local address 127.0.0.1:35159)
0927 15:11:37.388323 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:37.388340 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:37.388382 (+ 42us) server_negotiation.cc:407] Connection header received
0927 15:11:37.389180 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:37.389202 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:37.389605 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:37.389927 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:37.390308 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.391365 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.392630 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.393337 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.393466 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:37.394455 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:37.394460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:37.394475 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:37.394926 (+ 451us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:37.394987 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:37.395012 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:37.395161 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:37.395432 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 15:11:37.395596 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":135,"threads_started":1}
I20260927 15:11:37.398880 28348 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1446393417e0492ea11839200eac337f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35159 } }
I20260927 15:11:37.399181 28348 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:37.399245 28348 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1446393417e0492ea11839200eac337f, State: Initialized, Role: FOLLOWER
I20260927 15:11:37.399739 28348 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1446393417e0492ea11839200eac337f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35159 } }
I20260927 15:11:37.399844 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:37.399895 28348 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:37.399956 28348 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:37.400056 28348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:37.401330 28348 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1446393417e0492ea11839200eac337f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35159 } }
I20260927 15:11:37.401712 28348 leader_election.cc:302] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1446393417e0492ea11839200eac337f; no voters:
I20260927 15:11:37.402081 28348 leader_election.cc:288] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:37.402158 28357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:37.402426 28357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f [term 1 LEADER]: Becoming Leader. State: Replica: 1446393417e0492ea11839200eac337f, State: Running, Role: LEADER
I20260927 15:11:37.402833 28357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1446393417e0492ea11839200eac337f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35159 } }
I20260927 15:11:37.403654 28348 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:37.404657 28359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1446393417e0492ea11839200eac337f. Latest consensus state: current_term: 1 leader_uuid: "1446393417e0492ea11839200eac337f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1446393417e0492ea11839200eac337f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35159 } } }
I20260927 15:11:37.404661 28360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1446393417e0492ea11839200eac337f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1446393417e0492ea11839200eac337f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35159 } } }
I20260927 15:11:37.404894 28359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:37.404901 28360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:37.407400 28351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:37.402085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:40567 (local address 127.0.0.1:35159)
0927 15:11:37.402206 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:37.402211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:37.402508 (+ 297us) server_negotiation.cc:407] Connection header received
0927 15:11:37.402687 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:37.402691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:37.402743 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:37.402818 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:37.403159 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.403904 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.405001 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:37.405323 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:37.405423 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:37.406447 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:37.406451 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:37.406453 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:37.406467 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:37.406495 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:37.406508 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:37.406875 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:37.407227 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 15:11:37.407266 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 15:11:37.409682 28373 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:11:37.410367 28373 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:11:37.410671 28367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:37.411818 28367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:37.418879 28367 catalog_manager.cc:1347] Generated new cluster ID: e4560d34ddb842798f6663fd02b84e1b
I20260927 15:11:37.418955 28367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:37.429581 28367 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:37.430974 28367 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:37.442529 28367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1446393417e0492ea11839200eac337f: Generated new TSK 0
I20260927 15:11:37.443303 28367 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:37.451481 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 28280
W20260927 15:11:37.458104 28265 connection.cc:535] client connection to 127.0.0.1:35159 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35159): Connection reset by peer (error 104)
2026-09-27T15:11:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:37 dist-test-slave-hhpq krb5kdc[28379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:37 dist-test-slave-hhpq krb5kdc[28379](info): set up 2 sockets
Sep 27 15:11:37 dist-test-slave-hhpq krb5kdc[28379](info): commencing operation
krb5kdc: starting...
W20260927 15:11:38.424994 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:11:38 dist-test-slave-hhpq krb5kdc[28379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:11:38.548199 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46073
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:38.659585 28392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:38.659878 28392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:38.659920 28392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:38.664005 28392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:38.664124 28392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:38.664144 28392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:38.664160 28392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:38.664175 28392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:38.669629 28392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:38.670673 28392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:38.671674 28392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:38.680357 28400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:38.680359 28398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:38.680359 28397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:38.680456 28392 server_base.cc:1034] running on GCE node
I20260927 15:11:38.681304 28392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:38.682516 28392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:38.683704 28392 hybrid_clock.cc:648] HybridClock initialized: now 1790521898683669 us; error 51 us; skew 500 ppm
Sep 27 15:11:38 dist-test-slave-hhpq krb5kdc[28379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:38.686734 28392 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:11:38.687741 28392 webserver.cc:460] Webserver started at http://127.0.0.1:46249/ using document root <none> and password file <none>
I20260927 15:11:38.688314 28392 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:38.688385 28392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:38.688592 28392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:38.690766 28392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "096094c3b92b45fcb9b68ddfa3f8cff5"
format_stamp: "Formatted at 2026-09-27 15:11:38 on dist-test-slave-hhpq"
server_key: "c2736b6344d7108c2e6266d599fa6710"
server_key_iv: "26808074172f12fa7265450bd8ddfec4"
server_key_version: "encryptionkey@0"
I20260927 15:11:38.691341 28392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "096094c3b92b45fcb9b68ddfa3f8cff5"
format_stamp: "Formatted at 2026-09-27 15:11:38 on dist-test-slave-hhpq"
server_key: "c2736b6344d7108c2e6266d599fa6710"
server_key_iv: "26808074172f12fa7265450bd8ddfec4"
server_key_version: "encryptionkey@0"
I20260927 15:11:38.695703 28392 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:11:38.698570 28407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:38.700030 28392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:11:38.700166 28392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "096094c3b92b45fcb9b68ddfa3f8cff5"
format_stamp: "Formatted at 2026-09-27 15:11:38 on dist-test-slave-hhpq"
server_key: "c2736b6344d7108c2e6266d599fa6710"
server_key_iv: "26808074172f12fa7265450bd8ddfec4"
server_key_version: "encryptionkey@0"
I20260927 15:11:38.700337 28392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:38.705164 28392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:38.708382 28392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:38.708561 28392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:38.715207 28392 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46073
I20260927 15:11:38.715216 28459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46073 every 8 connection(s)
I20260927 15:11:38.716192 28392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:38.719012 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:38.724668 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 28392
I20260927 15:11:38.724781 28460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5: Bootstrap starting.
I20260927 15:11:38.724805 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:38.725111 7239 external_mini_cluster.cc:1428] Setting key e85941496efd3aa604484cffb3d04d3a
I20260927 15:11:38.727663 28460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:38.728417 28460 log.cc:824] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:38.730590 28460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5: No bootstrap required, opened a new log
I20260927 15:11:38.732645 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:38.726802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:33877 (local address 127.0.0.1:46073)
0927 15:11:38.727311 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:38.727330 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:38.727375 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:11:38.728094 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:38.728129 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:38.728459 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:38.728797 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:38.731134 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:38.731140 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:38.731158 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:38.731173 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:38.731247 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:38.731274 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:38.731390 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:38.731467 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 15:11:38.731651 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":147,"threads_started":1}
I20260927 15:11:38.734979 28460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096094c3b92b45fcb9b68ddfa3f8cff5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46073 } }
I20260927 15:11:38.735252 28460 raft_consensus.cc:374] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:38.735302 28460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 096094c3b92b45fcb9b68ddfa3f8cff5, State: Initialized, Role: FOLLOWER
I20260927 15:11:38.735711 28460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096094c3b92b45fcb9b68ddfa3f8cff5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46073 } }
I20260927 15:11:38.735788 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:38.735860 28460 raft_consensus.cc:388] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:38.735926 28460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:38.736014 28460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:38.737260 28460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096094c3b92b45fcb9b68ddfa3f8cff5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46073 } }
I20260927 15:11:38.737645 28460 leader_election.cc:302] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 096094c3b92b45fcb9b68ddfa3f8cff5; no voters:
I20260927 15:11:38.737968 28460 leader_election.cc:288] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:38.739423 28460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:38.739669 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:38.738130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:58173 (local address 127.0.0.1:46073)
0927 15:11:38.738251 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:38.738256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:38.738404 (+ 148us) server_negotiation.cc:407] Connection header received
0927 15:11:38.738526 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:38.738529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:38.738566 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:38.738631 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:38.739307 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:38.739311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:38.739313 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:38.739325 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:38.739352 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:38.739362 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:38.739436 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:38.739507 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 15:11:38.739549 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:11:38.738049 28469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:38.741070 28469 raft_consensus.cc:686] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5 [term 1 LEADER]: Becoming Leader. State: Replica: 096094c3b92b45fcb9b68ddfa3f8cff5, State: Running, Role: LEADER
I20260927 15:11:38.741436 28469 consensus_queue.cc:237] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096094c3b92b45fcb9b68ddfa3f8cff5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46073 } }
I20260927 15:11:38.743291 28479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 096094c3b92b45fcb9b68ddfa3f8cff5. Latest consensus state: current_term: 1 leader_uuid: "096094c3b92b45fcb9b68ddfa3f8cff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096094c3b92b45fcb9b68ddfa3f8cff5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46073 } } }
I20260927 15:11:38.743438 28479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:38.743314 28471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "096094c3b92b45fcb9b68ddfa3f8cff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096094c3b92b45fcb9b68ddfa3f8cff5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46073 } } }
I20260927 15:11:38.743961 28471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5 [sys.catalog]: This master's current role is: LEADER
W20260927 15:11:38.746011 28484 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:11:38.746106 28484 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:11:38.746183 28485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:38.747017 28485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:38.752533 28485 catalog_manager.cc:1347] Generated new cluster ID: 7dda850e94d04edda0d1527474a378b2
I20260927 15:11:38.752615 28485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:38.769271 28485 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:38.769681 28485 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:38.778167 28485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 096094c3b92b45fcb9b68ddfa3f8cff5: Generated new TSK 0
I20260927 15:11:38.778910 28485 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:38.789050 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 28392
2026-09-27T15:11:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:38 dist-test-slave-hhpq krb5kdc[28491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:38 dist-test-slave-hhpq krb5kdc[28491](info): set up 2 sockets
Sep 27 15:11:38 dist-test-slave-hhpq krb5kdc[28491](info): commencing operation
krb5kdc: starting...
W20260927 15:11:39.748955 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:11:39 dist-test-slave-hhpq krb5kdc[28491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:11:39.872802 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42383
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:39.985831 28504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:39.986124 28504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:39.986168 28504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:39.990274 28504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:39.990346 28504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:39.990366 28504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:39.990381 28504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:39.990396 28504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:39.995769 28504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42383
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:39.996915 28504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:39.997951 28504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:11:40.006774 28504 server_base.cc:1034] running on GCE node
W20260927 15:11:40.006677 28512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:40.006670 28509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:40.006677 28510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:40.007639 28504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:40.008925 28504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:40.010090 28504 hybrid_clock.cc:648] HybridClock initialized: now 1790521900010064 us; error 49 us; skew 500 ppm
Sep 27 15:11:40 dist-test-slave-hhpq krb5kdc[28491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:40.013015 28504 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:11:40.014015 28504 webserver.cc:460] Webserver started at http://127.0.0.1:34089/ using document root <none> and password file <none>
I20260927 15:11:40.014586 28504 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:40.014650 28504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:40.014853 28504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:40.017058 28504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "c1f7f368eb244bff97cea3451477a455"
format_stamp: "Formatted at 2026-09-27 15:11:40 on dist-test-slave-hhpq"
server_key: "fe4502009691868a618ed6070b7f014a"
server_key_iv: "84ebf02a0f2b3a93dfc93ac37bc3c5cf"
server_key_version: "encryptionkey@0"
I20260927 15:11:40.017629 28504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "c1f7f368eb244bff97cea3451477a455"
format_stamp: "Formatted at 2026-09-27 15:11:40 on dist-test-slave-hhpq"
server_key: "fe4502009691868a618ed6070b7f014a"
server_key_iv: "84ebf02a0f2b3a93dfc93ac37bc3c5cf"
server_key_version: "encryptionkey@0"
I20260927 15:11:40.021579 28504 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:11:40.024279 28519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:40.025774 28504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:40.025908 28504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "c1f7f368eb244bff97cea3451477a455"
format_stamp: "Formatted at 2026-09-27 15:11:40 on dist-test-slave-hhpq"
server_key: "fe4502009691868a618ed6070b7f014a"
server_key_iv: "84ebf02a0f2b3a93dfc93ac37bc3c5cf"
server_key_version: "encryptionkey@0"
I20260927 15:11:40.026024 28504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:40.035879 28504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:40.039341 28504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:40.039549 28504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:40.046618 28504 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42383
I20260927 15:11:40.046633 28571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42383 every 8 connection(s)
I20260927 15:11:40.047658 28504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:40.049311 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 28504
I20260927 15:11:40.049463 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:40.049767 7239 external_mini_cluster.cc:1428] Setting key d46f282abcbbaca04ba4fc2d21552b60
I20260927 15:11:40.051118 28572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:40.057983 28572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1f7f368eb244bff97cea3451477a455: Bootstrap starting.
Sep 27 15:11:40 dist-test-slave-hhpq krb5kdc[28491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:11:40.060722 28572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1f7f368eb244bff97cea3451477a455: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:40.061576 28572 log.cc:824] T 00000000000000000000000000000000 P c1f7f368eb244bff97cea3451477a455: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:40.063687 28572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1f7f368eb244bff97cea3451477a455: No bootstrap required, opened a new log
I20260927 15:11:40.065003 28575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:40.051248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:41233 (local address 127.0.0.1:42383)
0927 15:11:40.051673 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:40.051689 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:40.051726 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:11:40.052383 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:40.052400 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:40.052775 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:40.053124 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:40.054040 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:40.055172 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:40.056353 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:40.056900 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:40.057054 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:40.060041 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:40.060047 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:40.060062 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:40.060094 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:40.061835 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:40.062477 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:40.062482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:40.062488 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:40.062607 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:40.062964 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:40.062966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:40.062967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:40.063396 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:40.063601 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:40.063948 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 15:11:40.064182 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":122,"threads_started":1}
I20260927 15:11:40.067592 28572 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1f7f368eb244bff97cea3451477a455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f7f368eb244bff97cea3451477a455" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42383 } }
I20260927 15:11:40.067836 28572 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1f7f368eb244bff97cea3451477a455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:40.067893 28572 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1f7f368eb244bff97cea3451477a455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1f7f368eb244bff97cea3451477a455, State: Initialized, Role: FOLLOWER
I20260927 15:11:40.068145 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:40.068359 28572 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1f7f368eb244bff97cea3451477a455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f7f368eb244bff97cea3451477a455" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42383 } }
I20260927 15:11:40.068500 28572 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1f7f368eb244bff97cea3451477a455 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:40.068565 28572 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1f7f368eb244bff97cea3451477a455 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:40.068663 28572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1f7f368eb244bff97cea3451477a455 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:40.069718 28572 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1f7f368eb244bff97cea3451477a455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f7f368eb244bff97cea3451477a455" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42383 } }
I20260927 15:11:40.070094 28572 leader_election.cc:302] T 00000000000000000000000000000000 P c1f7f368eb244bff97cea3451477a455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1f7f368eb244bff97cea3451477a455; no voters:
I20260927 15:11:40.070410 28572 leader_election.cc:288] T 00000000000000000000000000000000 P c1f7f368eb244bff97cea3451477a455 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:40.070511 28582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1f7f368eb244bff97cea3451477a455 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:40.070730 28582 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1f7f368eb244bff97cea3451477a455 [term 1 LEADER]: Becoming Leader. State: Replica: c1f7f368eb244bff97cea3451477a455, State: Running, Role: LEADER
I20260927 15:11:40.071074 28582 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1f7f368eb244bff97cea3451477a455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f7f368eb244bff97cea3451477a455" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42383 } }
I20260927 15:11:40.071795 28572 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1f7f368eb244bff97cea3451477a455 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:40.072696 28583 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1f7f368eb244bff97cea3451477a455 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1f7f368eb244bff97cea3451477a455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f7f368eb244bff97cea3451477a455" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42383 } } }
I20260927 15:11:40.072820 28583 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1f7f368eb244bff97cea3451477a455 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:40.073211 28584 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1f7f368eb244bff97cea3451477a455 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1f7f368eb244bff97cea3451477a455. Latest consensus state: current_term: 1 leader_uuid: "c1f7f368eb244bff97cea3451477a455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f7f368eb244bff97cea3451477a455" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42383 } } }
I20260927 15:11:40.073366 28584 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1f7f368eb244bff97cea3451477a455 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:40.073688 28591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:40.076601 28591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:40.079510 28575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:40.070189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:37197 (local address 127.0.0.1:42383)
0927 15:11:40.070308 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:40.070313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:40.070543 (+ 230us) server_negotiation.cc:407] Connection header received
0927 15:11:40.070709 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:40.070712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:40.070763 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:40.070840 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:40.071757 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:40.072549 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:40.073593 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:40.073915 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:40.074011 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:40.075675 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:40.075679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:40.075682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:40.075710 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:40.077430 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:40.078058 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:40.078062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:40.078064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:40.078156 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:40.078570 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:40.078574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:40.078576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:40.078801 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:40.078945 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:40.079258 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 15:11:40.079361 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:11:40.082446 28591 catalog_manager.cc:1347] Generated new cluster ID: 4a508faa9bab4676a2e6438f6ce8865f
I20260927 15:11:40.082526 28591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:40.106035 28591 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:40.107393 28591 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:40.115410 28591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1f7f368eb244bff97cea3451477a455: Generated new TSK 0
I20260927 15:11:40.116346 28591 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:40.126571 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 28504
W20260927 15:11:40.131170 28489 connection.cc:535] client connection to 127.0.0.1:42383 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42383): Connection reset by peer (error 104)
2026-09-27T15:11:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:40 dist-test-slave-hhpq krb5kdc[28603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:40 dist-test-slave-hhpq krb5kdc[28603](info): set up 2 sockets
Sep 27 15:11:40 dist-test-slave-hhpq krb5kdc[28603](info): commencing operation
krb5kdc: starting...
W20260927 15:11:41.077069 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:11:41 dist-test-slave-hhpq krb5kdc[28603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:11:41.200523 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44713
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:41.311262 28616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:41.311553 28616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:41.311594 28616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:41.315888 28616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:41.315975 28616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:41.315994 28616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:41.316010 28616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:41.316025 28616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:41.321885 28616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44713
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:41.323031 28616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:41.324100 28616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:41.333204 28622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:41.333189 28624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:41.333170 28621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:41.333281 28616 server_base.cc:1034] running on GCE node
I20260927 15:11:41.334013 28616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:41.335206 28616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:41.336366 28616 hybrid_clock.cc:648] HybridClock initialized: now 1790521901336348 us; error 29 us; skew 500 ppm
Sep 27 15:11:41 dist-test-slave-hhpq krb5kdc[28603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:41.339279 28616 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:11:41.340288 28616 webserver.cc:460] Webserver started at http://127.0.0.1:35757/ using document root <none> and password file <none>
I20260927 15:11:41.340898 28616 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:41.340970 28616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:41.341164 28616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:41.343312 28616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "8b8e3b75c1004a1b972231aa30f7915d"
format_stamp: "Formatted at 2026-09-27 15:11:41 on dist-test-slave-hhpq"
server_key: "36e7a026c89216a8eaf1bb0d2c6584c4"
server_key_iv: "60ef69c01fc0bb9343b357b52cf73c4d"
server_key_version: "encryptionkey@0"
I20260927 15:11:41.343868 28616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "8b8e3b75c1004a1b972231aa30f7915d"
format_stamp: "Formatted at 2026-09-27 15:11:41 on dist-test-slave-hhpq"
server_key: "36e7a026c89216a8eaf1bb0d2c6584c4"
server_key_iv: "60ef69c01fc0bb9343b357b52cf73c4d"
server_key_version: "encryptionkey@0"
I20260927 15:11:41.347883 28616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:11:41.350768 28631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:41.352108 28616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:11:41.352241 28616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "8b8e3b75c1004a1b972231aa30f7915d"
format_stamp: "Formatted at 2026-09-27 15:11:41 on dist-test-slave-hhpq"
server_key: "36e7a026c89216a8eaf1bb0d2c6584c4"
server_key_iv: "60ef69c01fc0bb9343b357b52cf73c4d"
server_key_version: "encryptionkey@0"
I20260927 15:11:41.352346 28616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:41.364254 28616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:41.367364 28616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:41.367528 28616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:41.374210 28616 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44713
I20260927 15:11:41.374256 28683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44713 every 8 connection(s)
I20260927 15:11:41.375166 28616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:41.376684 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 28616
I20260927 15:11:41.376780 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:41.377099 7239 external_mini_cluster.cc:1428] Setting key 1ccd8a0ce2b83c82c0db9127064faeee
I20260927 15:11:41.378129 28684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:41.385116 28684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8e3b75c1004a1b972231aa30f7915d: Bootstrap starting.
I20260927 15:11:41.388262 28684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b8e3b75c1004a1b972231aa30f7915d: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:41.388657 28687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:41.378530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:44087 (local address 127.0.0.1:44713)
0927 15:11:41.378965 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:41.378984 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:41.379036 (+ 52us) server_negotiation.cc:407] Connection header received
0927 15:11:41.379843 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:41.379868 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:41.380332 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:41.380729 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:41.381222 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:41.382704 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:41.383721 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:41.384450 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:41.384608 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:41.385974 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:41.385981 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:41.386000 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:41.386664 (+ 664us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:41.386748 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:41.386784 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:41.386956 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:41.387262 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 15:11:41.387494 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":131,"threads_started":1}
I20260927 15:11:41.389348 28684 log.cc:824] T 00000000000000000000000000000000 P 8b8e3b75c1004a1b972231aa30f7915d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:41.392072 28684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8e3b75c1004a1b972231aa30f7915d: No bootstrap required, opened a new log
I20260927 15:11:41.392107 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:41.396680 28684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b8e3b75c1004a1b972231aa30f7915d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8e3b75c1004a1b972231aa30f7915d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44713 } }
I20260927 15:11:41.397042 28684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b8e3b75c1004a1b972231aa30f7915d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:41.397120 28684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b8e3b75c1004a1b972231aa30f7915d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8e3b75c1004a1b972231aa30f7915d, State: Initialized, Role: FOLLOWER
I20260927 15:11:41.397641 28684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b8e3b75c1004a1b972231aa30f7915d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8e3b75c1004a1b972231aa30f7915d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44713 } }
I20260927 15:11:41.397804 28684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b8e3b75c1004a1b972231aa30f7915d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:41.397858 28684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b8e3b75c1004a1b972231aa30f7915d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:41.397964 28684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b8e3b75c1004a1b972231aa30f7915d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:41.398485 28687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:41.394381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:36055 (local address 127.0.0.1:44713)
0927 15:11:41.394502 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:41.394507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:41.394573 (+ 66us) server_negotiation.cc:407] Connection header received
0927 15:11:41.394678 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:41.394681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:41.394728 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:41.394810 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:41.395036 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:41.395817 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:41.396611 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:41.396942 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:41.397056 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:41.397999 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:41.398003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:41.398006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:41.398022 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:41.398056 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:41.398071 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:41.398155 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:41.398308 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 15:11:41.398348 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:11:41.399168 28684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b8e3b75c1004a1b972231aa30f7915d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8e3b75c1004a1b972231aa30f7915d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44713 } }
I20260927 15:11:41.399569 28684 leader_election.cc:302] T 00000000000000000000000000000000 P 8b8e3b75c1004a1b972231aa30f7915d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b8e3b75c1004a1b972231aa30f7915d; no voters:
I20260927 15:11:41.399907 28684 leader_election.cc:288] T 00000000000000000000000000000000 P 8b8e3b75c1004a1b972231aa30f7915d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:41.400002 28694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b8e3b75c1004a1b972231aa30f7915d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:41.400216 28694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b8e3b75c1004a1b972231aa30f7915d [term 1 LEADER]: Becoming Leader. State: Replica: 8b8e3b75c1004a1b972231aa30f7915d, State: Running, Role: LEADER
I20260927 15:11:41.400524 28694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b8e3b75c1004a1b972231aa30f7915d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8e3b75c1004a1b972231aa30f7915d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44713 } }
I20260927 15:11:41.401216 28684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b8e3b75c1004a1b972231aa30f7915d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:41.402153 28695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b8e3b75c1004a1b972231aa30f7915d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b8e3b75c1004a1b972231aa30f7915d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8e3b75c1004a1b972231aa30f7915d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44713 } } }
I20260927 15:11:41.402283 28695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b8e3b75c1004a1b972231aa30f7915d [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:41.402530 28696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b8e3b75c1004a1b972231aa30f7915d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b8e3b75c1004a1b972231aa30f7915d. Latest consensus state: current_term: 1 leader_uuid: "8b8e3b75c1004a1b972231aa30f7915d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8e3b75c1004a1b972231aa30f7915d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44713 } } }
I20260927 15:11:41.402616 28696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b8e3b75c1004a1b972231aa30f7915d [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:41.402652 28703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:41.406191 28703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:41.412417 28703 catalog_manager.cc:1347] Generated new cluster ID: 388f03fc5b224f8eaf87e53e3d7f33cf
I20260927 15:11:41.412510 28703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:41.444078 28703 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:41.445379 28703 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:41.452206 28703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b8e3b75c1004a1b972231aa30f7915d: Generated new TSK 0
I20260927 15:11:41.453153 28703 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:41.458848 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 28616
W20260927 15:11:41.463531 28601 connection.cc:535] client connection to 127.0.0.1:44713 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44713): Connection reset by peer (error 104)
2026-09-27T15:11:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:41 dist-test-slave-hhpq krb5kdc[28715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:41 dist-test-slave-hhpq krb5kdc[28715](info): set up 2 sockets
Sep 27 15:11:41 dist-test-slave-hhpq krb5kdc[28715](info): commencing operation
krb5kdc: starting...
W20260927 15:11:42.409629 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:11:42 dist-test-slave-hhpq krb5kdc[28715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:11:42.536715 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43169
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:42.646591 28728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:42.646910 28728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:42.646955 28728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:42.651145 28728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:42.651232 28728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:42.651252 28728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:42.651269 28728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:42.651285 28728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:42.656883 28728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43169
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:42.658027 28728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:42.659075 28728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:42.667831 28734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:42.667831 28733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:42.668027 28728 server_base.cc:1034] running on GCE node
W20260927 15:11:42.667868 28736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:42.668740 28728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:42.669960 28728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:42.671141 28728 hybrid_clock.cc:648] HybridClock initialized: now 1790521902671110 us; error 43 us; skew 500 ppm
Sep 27 15:11:42 dist-test-slave-hhpq krb5kdc[28715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:42.677928 28728 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:11:42.679009 28728 webserver.cc:460] Webserver started at http://127.0.0.1:36253/ using document root <none> and password file <none>
I20260927 15:11:42.679590 28728 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:42.679653 28728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:42.679833 28728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:42.682112 28728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "79543df8a20c4f4d8afe4a42b458040d"
format_stamp: "Formatted at 2026-09-27 15:11:42 on dist-test-slave-hhpq"
server_key: "e180efacbf929260f7017493c12aecc3"
server_key_iv: "3b7a3a4c8fefc4166b9b979b86bf4392"
server_key_version: "encryptionkey@0"
I20260927 15:11:42.682683 28728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "79543df8a20c4f4d8afe4a42b458040d"
format_stamp: "Formatted at 2026-09-27 15:11:42 on dist-test-slave-hhpq"
server_key: "e180efacbf929260f7017493c12aecc3"
server_key_iv: "3b7a3a4c8fefc4166b9b979b86bf4392"
server_key_version: "encryptionkey@0"
I20260927 15:11:42.686743 28728 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:11:42.689481 28743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:42.690935 28728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:42.691095 28728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "79543df8a20c4f4d8afe4a42b458040d"
format_stamp: "Formatted at 2026-09-27 15:11:42 on dist-test-slave-hhpq"
server_key: "e180efacbf929260f7017493c12aecc3"
server_key_iv: "3b7a3a4c8fefc4166b9b979b86bf4392"
server_key_version: "encryptionkey@0"
I20260927 15:11:42.691203 28728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:42.696072 28728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:42.699317 28728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:42.699481 28728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:42.706452 28728 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43169
I20260927 15:11:42.706485 28795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43169 every 8 connection(s)
I20260927 15:11:42.707445 28728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:42.710539 28796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:42.713153 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 28728
I20260927 15:11:42.713264 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:42.713575 7239 external_mini_cluster.cc:1428] Setting key cbaac58695b8b84add2b5eb9eb00c6e9
I20260927 15:11:42.716992 28796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79543df8a20c4f4d8afe4a42b458040d: Bootstrap starting.
I20260927 15:11:42.718971 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:42.715015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:60669 (local address 127.0.0.1:43169)
0927 15:11:42.715501 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:42.715517 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:42.715556 (+ 39us) server_negotiation.cc:407] Connection header received
0927 15:11:42.716211 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:42.716230 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:42.716554 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:42.716998 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:42.717747 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:42.717751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:42.717764 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:42.717774 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:42.717831 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:42.717850 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:42.717987 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:42.718061 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 15:11:42.718230 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":166,"threads_started":1}
I20260927 15:11:42.719625 28796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79543df8a20c4f4d8afe4a42b458040d: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:42.720479 28796 log.cc:824] T 00000000000000000000000000000000 P 79543df8a20c4f4d8afe4a42b458040d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:42.722080 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:42.722560 28796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79543df8a20c4f4d8afe4a42b458040d: No bootstrap required, opened a new log
I20260927 15:11:42.725710 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:42.724082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:32821 (local address 127.0.0.1:43169)
0927 15:11:42.724208 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:42.724212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:42.724386 (+ 174us) server_negotiation.cc:407] Connection header received
0927 15:11:42.724512 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:42.724516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:42.724562 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:42.724629 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:42.725326 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:42.725330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:42.725333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:42.725352 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:42.725381 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:42.725391 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:42.725467 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:42.725540 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 15:11:42.725584 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 15:11:42.726606 28796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79543df8a20c4f4d8afe4a42b458040d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79543df8a20c4f4d8afe4a42b458040d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43169 } }
I20260927 15:11:42.726898 28796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79543df8a20c4f4d8afe4a42b458040d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:42.726946 28796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79543df8a20c4f4d8afe4a42b458040d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79543df8a20c4f4d8afe4a42b458040d, State: Initialized, Role: FOLLOWER
I20260927 15:11:42.727386 28796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79543df8a20c4f4d8afe4a42b458040d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79543df8a20c4f4d8afe4a42b458040d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43169 } }
I20260927 15:11:42.727523 28796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79543df8a20c4f4d8afe4a42b458040d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:42.727566 28796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79543df8a20c4f4d8afe4a42b458040d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:42.727638 28796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79543df8a20c4f4d8afe4a42b458040d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:42.728633 28796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79543df8a20c4f4d8afe4a42b458040d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79543df8a20c4f4d8afe4a42b458040d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43169 } }
I20260927 15:11:42.729025 28796 leader_election.cc:302] T 00000000000000000000000000000000 P 79543df8a20c4f4d8afe4a42b458040d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79543df8a20c4f4d8afe4a42b458040d; no voters:
I20260927 15:11:42.729382 28796 leader_election.cc:288] T 00000000000000000000000000000000 P 79543df8a20c4f4d8afe4a42b458040d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:42.729477 28806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79543df8a20c4f4d8afe4a42b458040d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:42.729696 28806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79543df8a20c4f4d8afe4a42b458040d [term 1 LEADER]: Becoming Leader. State: Replica: 79543df8a20c4f4d8afe4a42b458040d, State: Running, Role: LEADER
I20260927 15:11:42.730059 28806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79543df8a20c4f4d8afe4a42b458040d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79543df8a20c4f4d8afe4a42b458040d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43169 } }
I20260927 15:11:42.730721 28796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79543df8a20c4f4d8afe4a42b458040d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:42.731710 28807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79543df8a20c4f4d8afe4a42b458040d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79543df8a20c4f4d8afe4a42b458040d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79543df8a20c4f4d8afe4a42b458040d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43169 } } }
I20260927 15:11:42.731849 28807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79543df8a20c4f4d8afe4a42b458040d [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:42.732151 28808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79543df8a20c4f4d8afe4a42b458040d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79543df8a20c4f4d8afe4a42b458040d. Latest consensus state: current_term: 1 leader_uuid: "79543df8a20c4f4d8afe4a42b458040d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79543df8a20c4f4d8afe4a42b458040d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43169 } } }
I20260927 15:11:42.732242 28808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79543df8a20c4f4d8afe4a42b458040d [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:42.732331 28815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:42.735780 28815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:42.742307 28815 catalog_manager.cc:1347] Generated new cluster ID: aa8525e6acec4d0c8c2d2572b2ffe691
I20260927 15:11:42.742413 28815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:42.763731 28815 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:42.764115 28815 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:42.770735 28815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79543df8a20c4f4d8afe4a42b458040d: Generated new TSK 0
I20260927 15:11:42.771587 28815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:42.775273 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 28728
2026-09-27T15:11:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:42 dist-test-slave-hhpq krb5kdc[28827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:42 dist-test-slave-hhpq krb5kdc[28827](info): set up 2 sockets
Sep 27 15:11:42 dist-test-slave-hhpq krb5kdc[28827](info): commencing operation
krb5kdc: starting...
W20260927 15:11:43.728941 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:11:43 dist-test-slave-hhpq krb5kdc[28827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:11:43.847863 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:38785
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:38785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:43.961905 28840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:43.962218 28840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:43.962261 28840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:43.966463 28840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:43.966549 28840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:43.966567 28840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:43.966583 28840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:43.966598 28840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:43.972383 28840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:38785
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:38785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:43.973608 28840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:43.974664 28840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:43.983513 28845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:43.983629 28840 server_base.cc:1034] running on GCE node
W20260927 15:11:43.983517 28848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:43.983513 28846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:43.984549 28840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:43.985740 28840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:43.986910 28840 hybrid_clock.cc:648] HybridClock initialized: now 1790521903986890 us; error 36 us; skew 500 ppm
Sep 27 15:11:43 dist-test-slave-hhpq krb5kdc[28827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521903, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:43.989926 28840 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:11:43.990895 28840 webserver.cc:460] Webserver started at http://127.7.17.254:40205/ using document root <none> and password file <none>
I20260927 15:11:43.991464 28840 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:43.991531 28840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:43.991743 28840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:43.994122 28840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "d8cc88508ecb4d25932039bc56b9bd56"
format_stamp: "Formatted at 2026-09-27 15:11:43 on dist-test-slave-hhpq"
server_key: "d47ec3e5a04b229add0495c561c0c7ac"
server_key_iv: "0f26f2653b8d6c849cdb46a12ca29c8f"
server_key_version: "encryptionkey@0"
I20260927 15:11:43.994719 28840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "d8cc88508ecb4d25932039bc56b9bd56"
format_stamp: "Formatted at 2026-09-27 15:11:43 on dist-test-slave-hhpq"
server_key: "d47ec3e5a04b229add0495c561c0c7ac"
server_key_iv: "0f26f2653b8d6c849cdb46a12ca29c8f"
server_key_version: "encryptionkey@0"
I20260927 15:11:43.998778 28840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:11:44.001506 28855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:44.002960 28840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:44.003134 28840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "d8cc88508ecb4d25932039bc56b9bd56"
format_stamp: "Formatted at 2026-09-27 15:11:43 on dist-test-slave-hhpq"
server_key: "d47ec3e5a04b229add0495c561c0c7ac"
server_key_iv: "0f26f2653b8d6c849cdb46a12ca29c8f"
server_key_version: "encryptionkey@0"
I20260927 15:11:44.003257 28840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:44.016115 28840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:44.019349 28840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:44.019555 28840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:44.026660 28840 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:38785
I20260927 15:11:44.026754 28907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:38785 every 8 connection(s)
I20260927 15:11:44.027670 28840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:44.030547 28908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:44.034170 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 28840
I20260927 15:11:44.034304 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:44.034552 7239 external_mini_cluster.cc:1428] Setting key fe54e9cf8a6108b0f72ebfef4beaed86
I20260927 15:11:44.037025 28908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8cc88508ecb4d25932039bc56b9bd56: Bootstrap starting.
I20260927 15:11:44.039852 28908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8cc88508ecb4d25932039bc56b9bd56: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:44.040755 28908 log.cc:824] T 00000000000000000000000000000000 P d8cc88508ecb4d25932039bc56b9bd56: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:44.043109 28908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8cc88508ecb4d25932039bc56b9bd56: No bootstrap required, opened a new log
Sep 27 15:11:44 dist-test-slave-hhpq krb5kdc[28827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:11:44.047592 28908 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8cc88508ecb4d25932039bc56b9bd56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8cc88508ecb4d25932039bc56b9bd56" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38785 } }
I20260927 15:11:44.047883 28908 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8cc88508ecb4d25932039bc56b9bd56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:44.047945 28908 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8cc88508ecb4d25932039bc56b9bd56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8cc88508ecb4d25932039bc56b9bd56, State: Initialized, Role: FOLLOWER
I20260927 15:11:44.048382 28908 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8cc88508ecb4d25932039bc56b9bd56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8cc88508ecb4d25932039bc56b9bd56" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38785 } }
I20260927 15:11:44.048502 28908 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8cc88508ecb4d25932039bc56b9bd56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:44.048565 28908 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8cc88508ecb4d25932039bc56b9bd56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:44.048664 28908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8cc88508ecb4d25932039bc56b9bd56 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:44.049739 28908 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8cc88508ecb4d25932039bc56b9bd56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8cc88508ecb4d25932039bc56b9bd56" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38785 } }
I20260927 15:11:44.050107 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:44.036016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53235 (local address 127.7.17.254:38785)
0927 15:11:44.036487 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:44.036506 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:44.036553 (+ 47us) server_negotiation.cc:407] Connection header received
0927 15:11:44.037457 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:44.037484 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:44.037961 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:44.038354 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:44.039044 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.040510 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.041618 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.042284 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.042440 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:44.045054 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:44.045060 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:44.045078 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:44.045110 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:44.047011 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:44.047503 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:44.047508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:44.047514 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:44.047657 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:44.047947 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:44.047951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:44.047952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:44.048435 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:44.048662 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:44.049021 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 15:11:44.049266 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":122,"threads_started":1}
I20260927 15:11:44.050094 28908 leader_election.cc:302] T 00000000000000000000000000000000 P d8cc88508ecb4d25932039bc56b9bd56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8cc88508ecb4d25932039bc56b9bd56; no voters:
I20260927 15:11:44.050477 28908 leader_election.cc:288] T 00000000000000000000000000000000 P d8cc88508ecb4d25932039bc56b9bd56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:44.050578 28913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8cc88508ecb4d25932039bc56b9bd56 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:44.050782 28913 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8cc88508ecb4d25932039bc56b9bd56 [term 1 LEADER]: Becoming Leader. State: Replica: d8cc88508ecb4d25932039bc56b9bd56, State: Running, Role: LEADER
I20260927 15:11:44.051126 28913 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8cc88508ecb4d25932039bc56b9bd56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8cc88508ecb4d25932039bc56b9bd56" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38785 } }
I20260927 15:11:44.051898 28908 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8cc88508ecb4d25932039bc56b9bd56 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:44.052814 28914 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8cc88508ecb4d25932039bc56b9bd56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8cc88508ecb4d25932039bc56b9bd56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8cc88508ecb4d25932039bc56b9bd56" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38785 } } }
I20260927 15:11:44.053049 28914 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8cc88508ecb4d25932039bc56b9bd56 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:44.053506 28922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:44.053496 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:44.055518 28915 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8cc88508ecb4d25932039bc56b9bd56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8cc88508ecb4d25932039bc56b9bd56. Latest consensus state: current_term: 1 leader_uuid: "d8cc88508ecb4d25932039bc56b9bd56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8cc88508ecb4d25932039bc56b9bd56" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38785 } } }
I20260927 15:11:44.055631 28915 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8cc88508ecb4d25932039bc56b9bd56 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:44.057824 28922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:44.066056 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:44.055841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46515 (local address 127.7.17.254:38785)
0927 15:11:44.055956 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:44.055961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:44.057135 (+ 1174us) server_negotiation.cc:407] Connection header received
0927 15:11:44.057271 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:44.057274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:44.057325 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:44.057413 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:44.058341 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.059143 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.060387 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:44.060749 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:44.060854 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:44.062773 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:44.062776 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:44.062778 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:44.062803 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:44.064175 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:44.064740 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:44.064742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:44.064743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:44.064817 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:44.065216 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:44.065219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:44.065220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:44.065402 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:44.065541 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:44.065799 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 15:11:44.065894 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:11:44.066763 28922 catalog_manager.cc:1347] Generated new cluster ID: 45f90a03fade4ae08a242bd220d80288
I20260927 15:11:44.066838 28922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:44.075491 28922 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:44.076650 28922 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:44.084045 28922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8cc88508ecb4d25932039bc56b9bd56: Generated new TSK 0
I20260927 15:11:44.085027 28922 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:44.092455 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 28840
W20260927 15:11:44.097172 28825 connection.cc:535] client connection to 127.7.17.254:38785 recv error: Network error: failed to read from TLS socket (remote: 127.7.17.254:38785): Connection reset by peer (error 104)
2026-09-27T15:11:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:44 dist-test-slave-hhpq krb5kdc[28939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:44 dist-test-slave-hhpq krb5kdc[28939](info): set up 2 sockets
Sep 27 15:11:44 dist-test-slave-hhpq krb5kdc[28939](info): commencing operation
krb5kdc: starting...
W20260927 15:11:45.048813 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:11:45 dist-test-slave-hhpq krb5kdc[28939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:11:45.172300 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:34521
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:34521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:45.281764 28952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:45.282048 28952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:45.282089 28952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:45.286497 28952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:45.286568 28952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:45.286587 28952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:45.286603 28952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:45.286617 28952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:45.292100 28952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:34521
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:34521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:45.293252 28952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:45.294268 28952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:45.303186 28960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:45.303205 28957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:45.303200 28958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:45.303540 28952 server_base.cc:1034] running on GCE node
I20260927 15:11:45.304138 28952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:45.305387 28952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:45.306550 28952 hybrid_clock.cc:648] HybridClock initialized: now 1790521905306535 us; error 32 us; skew 500 ppm
Sep 27 15:11:45 dist-test-slave-hhpq krb5kdc[28939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521905, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:45.309348 28952 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:11:45.310307 28952 webserver.cc:460] Webserver started at http://127.7.17.254:44259/ using document root <none> and password file <none>
I20260927 15:11:45.310864 28952 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:45.310932 28952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:45.311134 28952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:45.313243 28952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "ec3f46c3d7974b87b30c48b2074336f0"
format_stamp: "Formatted at 2026-09-27 15:11:45 on dist-test-slave-hhpq"
server_key: "a51e255678eba9f11100e9e8aa50f4e4"
server_key_iv: "0c1168aa0c575bf18be6752477c29484"
server_key_version: "encryptionkey@0"
I20260927 15:11:45.313795 28952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "ec3f46c3d7974b87b30c48b2074336f0"
format_stamp: "Formatted at 2026-09-27 15:11:45 on dist-test-slave-hhpq"
server_key: "a51e255678eba9f11100e9e8aa50f4e4"
server_key_iv: "0c1168aa0c575bf18be6752477c29484"
server_key_version: "encryptionkey@0"
I20260927 15:11:45.317629 28952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:11:45.320314 28967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:45.321697 28952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:11:45.321830 28952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "ec3f46c3d7974b87b30c48b2074336f0"
format_stamp: "Formatted at 2026-09-27 15:11:45 on dist-test-slave-hhpq"
server_key: "a51e255678eba9f11100e9e8aa50f4e4"
server_key_iv: "0c1168aa0c575bf18be6752477c29484"
server_key_version: "encryptionkey@0"
I20260927 15:11:45.321945 28952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:45.338795 28952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:45.342216 28952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:45.342424 28952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:45.349354 28952 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:34521
I20260927 15:11:45.349388 29019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:34521 every 8 connection(s)
I20260927 15:11:45.350376 28952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:45.353359 29020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:45.358712 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 28952
I20260927 15:11:45.358867 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:45.359100 7239 external_mini_cluster.cc:1428] Setting key 8f340f7c52c183db3b2ac3c2807adece
I20260927 15:11:45.359207 29020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec3f46c3d7974b87b30c48b2074336f0: Bootstrap starting.
I20260927 15:11:45.362134 29020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec3f46c3d7974b87b30c48b2074336f0: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:45.363066 29020 log.cc:824] T 00000000000000000000000000000000 P ec3f46c3d7974b87b30c48b2074336f0: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:45.365530 29020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec3f46c3d7974b87b30c48b2074336f0: No bootstrap required, opened a new log
I20260927 15:11:45.369792 29023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:45.360541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45905 (local address 127.7.17.254:34521)
0927 15:11:45.361034 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:45.361053 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:45.361099 (+ 46us) server_negotiation.cc:407] Connection header received
0927 15:11:45.361986 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:45.362012 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:45.362532 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:45.362933 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:45.363379 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:45.364706 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:45.365839 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:45.366561 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:45.366736 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:45.367620 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:45.367627 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:45.367647 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:45.368173 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:45.368246 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:45.368277 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:45.368412 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:45.368565 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 15:11:45.368726 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":126,"threads_started":1}
I20260927 15:11:45.370267 29020 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec3f46c3d7974b87b30c48b2074336f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3f46c3d7974b87b30c48b2074336f0" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 34521 } }
I20260927 15:11:45.370501 29020 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec3f46c3d7974b87b30c48b2074336f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:45.370537 29020 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec3f46c3d7974b87b30c48b2074336f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3f46c3d7974b87b30c48b2074336f0, State: Initialized, Role: FOLLOWER
I20260927 15:11:45.371011 29020 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec3f46c3d7974b87b30c48b2074336f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3f46c3d7974b87b30c48b2074336f0" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 34521 } }
I20260927 15:11:45.371156 29020 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec3f46c3d7974b87b30c48b2074336f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:45.371225 29020 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec3f46c3d7974b87b30c48b2074336f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:45.371322 29020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec3f46c3d7974b87b30c48b2074336f0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:45.372387 29020 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec3f46c3d7974b87b30c48b2074336f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3f46c3d7974b87b30c48b2074336f0" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 34521 } }
I20260927 15:11:45.372717 29020 leader_election.cc:302] T 00000000000000000000000000000000 P ec3f46c3d7974b87b30c48b2074336f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec3f46c3d7974b87b30c48b2074336f0; no voters:
I20260927 15:11:45.372902 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:45.373104 29020 leader_election.cc:288] T 00000000000000000000000000000000 P ec3f46c3d7974b87b30c48b2074336f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:45.373190 29025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec3f46c3d7974b87b30c48b2074336f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:45.373385 29025 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec3f46c3d7974b87b30c48b2074336f0 [term 1 LEADER]: Becoming Leader. State: Replica: ec3f46c3d7974b87b30c48b2074336f0, State: Running, Role: LEADER
I20260927 15:11:45.373796 29025 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec3f46c3d7974b87b30c48b2074336f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3f46c3d7974b87b30c48b2074336f0" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 34521 } }
I20260927 15:11:45.374294 29020 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec3f46c3d7974b87b30c48b2074336f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:45.375116 29025 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec3f46c3d7974b87b30c48b2074336f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec3f46c3d7974b87b30c48b2074336f0. Latest consensus state: current_term: 1 leader_uuid: "ec3f46c3d7974b87b30c48b2074336f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3f46c3d7974b87b30c48b2074336f0" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 34521 } } }
I20260927 15:11:45.375294 29025 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec3f46c3d7974b87b30c48b2074336f0 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:45.375523 29026 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec3f46c3d7974b87b30c48b2074336f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec3f46c3d7974b87b30c48b2074336f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3f46c3d7974b87b30c48b2074336f0" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 34521 } } }
I20260927 15:11:45.375630 29026 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec3f46c3d7974b87b30c48b2074336f0 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:45.375728 29038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:45.379034 29038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:45.385004 29023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:45.375197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49635 (local address 127.7.17.254:34521)
0927 15:11:45.375308 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:45.375312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:45.381125 (+ 5813us) server_negotiation.cc:407] Connection header received
0927 15:11:45.381262 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:45.381265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:45.381315 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:45.381381 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:45.381693 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:45.382230 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:45.383059 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:45.383355 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:45.383464 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:45.384477 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:45.384481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:45.384484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:45.384500 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:45.384533 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:45.384547 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:45.384629 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:45.384784 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 15:11:45.384853 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:11:45.385954 29038 catalog_manager.cc:1347] Generated new cluster ID: e721c7cc101246709d7092373cb83caf
I20260927 15:11:45.386086 29038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:45.417372 29038 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:45.418568 29038 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:45.428423 29038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec3f46c3d7974b87b30c48b2074336f0: Generated new TSK 0
I20260927 15:11:45.429195 29038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:45.443887 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 28952
W20260927 15:11:45.448912 28937 connection.cc:535] client connection to 127.7.17.254:34521 recv error: Network error: failed to read from TLS socket (remote: 127.7.17.254:34521): Connection reset by peer (error 104)
2026-09-27T15:11:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:45 dist-test-slave-hhpq krb5kdc[29051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:45 dist-test-slave-hhpq krb5kdc[29051](info): set up 2 sockets
Sep 27 15:11:45 dist-test-slave-hhpq krb5kdc[29051](info): commencing operation
krb5kdc: starting...
W20260927 15:11:46.401836 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:11:46 dist-test-slave-hhpq krb5kdc[29051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:11:46.537518 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:39145
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:38053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:39145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:46.651559 29064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:46.651862 29064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:46.651904 29064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:46.656129 29064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:46.656220 29064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:46.656240 29064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:46.656258 29064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:46.656272 29064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:46.661978 29064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:38053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:39145
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:39145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:46.663128 29064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:46.664198 29064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:46.673177 29072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:46.673135 29070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:46.673170 29069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:46.673584 29064 server_base.cc:1034] running on GCE node
I20260927 15:11:46.674186 29064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:46.675374 29064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:46.676565 29064 hybrid_clock.cc:648] HybridClock initialized: now 1790521906676537 us; error 43 us; skew 500 ppm
Sep 27 15:11:46 dist-test-slave-hhpq krb5kdc[29051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521906, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:46.679673 29064 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:11:46.680694 29064 webserver.cc:460] Webserver started at http://127.7.17.254:36569/ using document root <none> and password file <none>
I20260927 15:11:46.681373 29064 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:46.681445 29064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:46.681663 29064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:46.683990 29064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "8063764686f74a79b3f5f4d5386e12ba"
format_stamp: "Formatted at 2026-09-27 15:11:46 on dist-test-slave-hhpq"
server_key: "716ea5f70464d6fe8550ef1bfa7a7e56"
server_key_iv: "e2dfc05b539ea890bd14865f123e5bfe"
server_key_version: "encryptionkey@0"
I20260927 15:11:46.684602 29064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "8063764686f74a79b3f5f4d5386e12ba"
format_stamp: "Formatted at 2026-09-27 15:11:46 on dist-test-slave-hhpq"
server_key: "716ea5f70464d6fe8550ef1bfa7a7e56"
server_key_iv: "e2dfc05b539ea890bd14865f123e5bfe"
server_key_version: "encryptionkey@0"
I20260927 15:11:46.688956 29064 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:11:46.691771 29079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:46.693424 29064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:46.693591 29064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "8063764686f74a79b3f5f4d5386e12ba"
format_stamp: "Formatted at 2026-09-27 15:11:46 on dist-test-slave-hhpq"
server_key: "716ea5f70464d6fe8550ef1bfa7a7e56"
server_key_iv: "e2dfc05b539ea890bd14865f123e5bfe"
server_key_version: "encryptionkey@0"
I20260927 15:11:46.693732 29064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:46.698585 29064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:46.701777 29064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:46.701962 29064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:46.708606 29064 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:39145
I20260927 15:11:46.708639 29131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:39145 every 8 connection(s)
I20260927 15:11:46.709560 29064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:46.712294 29132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:46.714087 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 29064
I20260927 15:11:46.714237 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:46.714525 7239 external_mini_cluster.cc:1428] Setting key 5b448fdd2e4efcd4af7ac531d050547c
I20260927 15:11:46.719252 29132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8063764686f74a79b3f5f4d5386e12ba: Bootstrap starting.
I20260927 15:11:46.720301 29135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:46.715917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57047 (local address 127.7.17.254:39145)
0927 15:11:46.716371 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:46.716390 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:46.716434 (+ 44us) server_negotiation.cc:407] Connection header received
0927 15:11:46.717280 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:46.717305 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:46.717718 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:46.718107 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:46.718835 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:46.718840 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:46.718855 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:46.718867 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:46.718931 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:46.718957 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:46.719099 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:46.719223 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 15:11:46.719385 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":147,"threads_started":1}
I20260927 15:11:46.721812 29132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8063764686f74a79b3f5f4d5386e12ba: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:46.722584 29132 log.cc:824] T 00000000000000000000000000000000 P 8063764686f74a79b3f5f4d5386e12ba: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:46.723313 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:46.724614 29132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8063764686f74a79b3f5f4d5386e12ba: No bootstrap required, opened a new log
I20260927 15:11:46.726894 29135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:46.725265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39981 (local address 127.7.17.254:39145)
0927 15:11:46.725369 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:46.725373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:46.725564 (+ 191us) server_negotiation.cc:407] Connection header received
0927 15:11:46.725674 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:46.725677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:46.725711 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:46.725775 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:46.726523 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:46.726527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:46.726529 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:46.726541 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:46.726567 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:46.726578 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:46.726652 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:46.726720 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 15:11:46.726774 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 15:11:46.729205 29132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8063764686f74a79b3f5f4d5386e12ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8063764686f74a79b3f5f4d5386e12ba" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39145 } }
I20260927 15:11:46.729542 29132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8063764686f74a79b3f5f4d5386e12ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:46.729615 29132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8063764686f74a79b3f5f4d5386e12ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8063764686f74a79b3f5f4d5386e12ba, State: Initialized, Role: FOLLOWER
I20260927 15:11:46.730105 29132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8063764686f74a79b3f5f4d5386e12ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8063764686f74a79b3f5f4d5386e12ba" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39145 } }
I20260927 15:11:46.730234 29132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8063764686f74a79b3f5f4d5386e12ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:46.730281 29132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8063764686f74a79b3f5f4d5386e12ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:46.730362 29132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8063764686f74a79b3f5f4d5386e12ba [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:46.731295 29132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8063764686f74a79b3f5f4d5386e12ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8063764686f74a79b3f5f4d5386e12ba" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39145 } }
I20260927 15:11:46.731637 29132 leader_election.cc:302] T 00000000000000000000000000000000 P 8063764686f74a79b3f5f4d5386e12ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8063764686f74a79b3f5f4d5386e12ba; no voters:
I20260927 15:11:46.731935 29132 leader_election.cc:288] T 00000000000000000000000000000000 P 8063764686f74a79b3f5f4d5386e12ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:46.732048 29142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8063764686f74a79b3f5f4d5386e12ba [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:46.732257 29142 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8063764686f74a79b3f5f4d5386e12ba [term 1 LEADER]: Becoming Leader. State: Replica: 8063764686f74a79b3f5f4d5386e12ba, State: Running, Role: LEADER
I20260927 15:11:46.732584 29142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8063764686f74a79b3f5f4d5386e12ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8063764686f74a79b3f5f4d5386e12ba" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39145 } }
I20260927 15:11:46.733300 29132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8063764686f74a79b3f5f4d5386e12ba [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:46.734356 29143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8063764686f74a79b3f5f4d5386e12ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8063764686f74a79b3f5f4d5386e12ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8063764686f74a79b3f5f4d5386e12ba" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39145 } } }
I20260927 15:11:46.734491 29143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8063764686f74a79b3f5f4d5386e12ba [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:46.734782 29144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8063764686f74a79b3f5f4d5386e12ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8063764686f74a79b3f5f4d5386e12ba. Latest consensus state: current_term: 1 leader_uuid: "8063764686f74a79b3f5f4d5386e12ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8063764686f74a79b3f5f4d5386e12ba" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 39145 } } }
I20260927 15:11:46.734889 29144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8063764686f74a79b3f5f4d5386e12ba [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:46.735778 29151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:46.738720 29151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:46.744645 29151 catalog_manager.cc:1347] Generated new cluster ID: 3e4c88f989854f31844d24bc910987b7
I20260927 15:11:46.744735 29151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:46.755788 29151 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:46.756204 29151 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:46.765643 29151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8063764686f74a79b3f5f4d5386e12ba: Generated new TSK 0
I20260927 15:11:46.766568 29151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:46.772755 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 29064
2026-09-27T15:11:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:46 dist-test-slave-hhpq krb5kdc[29163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:46 dist-test-slave-hhpq krb5kdc[29163](info): set up 2 sockets
Sep 27 15:11:46 dist-test-slave-hhpq krb5kdc[29163](info): commencing operation
krb5kdc: starting...
W20260927 15:11:47.716887 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:11:47 dist-test-slave-hhpq krb5kdc[29163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:11:47.839974 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:38395
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:44743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:38395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:47.951028 29176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:47.951328 29176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:47.951390 29176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:47.955555 29176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:47.955633 29176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:47.955662 29176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:47.955684 29176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:47.955713 29176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:47.961361 29176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:44743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:38395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:38395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:47.962519 29176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:47.963604 29176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:47.972347 29181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:47.972532 29176 server_base.cc:1034] running on GCE node
W20260927 15:11:47.972347 29184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:47.972373 29182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:47.973255 29176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:47.974429 29176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:47.975584 29176 hybrid_clock.cc:648] HybridClock initialized: now 1790521907975556 us; error 38 us; skew 500 ppm
Sep 27 15:11:47 dist-test-slave-hhpq krb5kdc[29163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521907, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:47.985442 29176 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:11:47.986500 29176 webserver.cc:460] Webserver started at http://127.7.17.254:46815/ using document root <none> and password file <none>
I20260927 15:11:47.987092 29176 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:47.987156 29176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:47.987347 29176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:47.989614 29176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "aad26f48b0cf4eb9a2bc22c970c9cfa3"
format_stamp: "Formatted at 2026-09-27 15:11:47 on dist-test-slave-hhpq"
server_key: "2231be5e5be8dd8e8873bd580c9287e9"
server_key_iv: "ec58338cc5c5055110adba09e3aec075"
server_key_version: "encryptionkey@0"
I20260927 15:11:47.990151 29176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "aad26f48b0cf4eb9a2bc22c970c9cfa3"
format_stamp: "Formatted at 2026-09-27 15:11:47 on dist-test-slave-hhpq"
server_key: "2231be5e5be8dd8e8873bd580c9287e9"
server_key_iv: "ec58338cc5c5055110adba09e3aec075"
server_key_version: "encryptionkey@0"
I20260927 15:11:47.994127 29176 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 15:11:47.996785 29191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:47.998242 29176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:47.998409 29176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "aad26f48b0cf4eb9a2bc22c970c9cfa3"
format_stamp: "Formatted at 2026-09-27 15:11:47 on dist-test-slave-hhpq"
server_key: "2231be5e5be8dd8e8873bd580c9287e9"
server_key_iv: "ec58338cc5c5055110adba09e3aec075"
server_key_version: "encryptionkey@0"
I20260927 15:11:47.998529 29176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:48.011757 29176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:48.015079 29176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:48.015262 29176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:48.021925 29176 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:38395
I20260927 15:11:48.021966 29243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:38395 every 8 connection(s)
I20260927 15:11:48.022881 29176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:48.025731 29244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:48.026458 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 29176
I20260927 15:11:48.026600 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:48.026919 7239 external_mini_cluster.cc:1428] Setting key 081b947471c2f7a4a259977226b8adc3
I20260927 15:11:48.032063 29244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad26f48b0cf4eb9a2bc22c970c9cfa3: Bootstrap starting.
I20260927 15:11:48.034768 29244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aad26f48b0cf4eb9a2bc22c970c9cfa3: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:48.035663 29244 log.cc:824] T 00000000000000000000000000000000 P aad26f48b0cf4eb9a2bc22c970c9cfa3: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:11:48 dist-test-slave-hhpq krb5kdc[29163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:11:48.037642 29244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad26f48b0cf4eb9a2bc22c970c9cfa3: No bootstrap required, opened a new log
I20260927 15:11:48.041260 29247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:48.028276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39227 (local address 127.7.17.254:38395)
0927 15:11:48.028689 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:48.028704 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:48.028740 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:11:48.029437 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:48.029454 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:48.029824 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:48.030109 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:48.030994 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:48.032023 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:48.033230 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:48.033711 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:48.033834 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:48.036605 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:48.036611 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:48.036627 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:48.036655 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:48.038351 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:48.038786 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:48.038791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:48.038796 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:48.038923 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:48.039185 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:48.039189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:48.039190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:48.039609 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:48.039792 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:48.040103 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 15:11:48.040427 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":124,"threads_started":1}
I20260927 15:11:48.041256 29244 raft_consensus.cc:348] T 00000000000000000000000000000000 P aad26f48b0cf4eb9a2bc22c970c9cfa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad26f48b0cf4eb9a2bc22c970c9cfa3" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38395 } }
I20260927 15:11:48.041575 29244 raft_consensus.cc:374] T 00000000000000000000000000000000 P aad26f48b0cf4eb9a2bc22c970c9cfa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:48.041631 29244 raft_consensus.cc:729] T 00000000000000000000000000000000 P aad26f48b0cf4eb9a2bc22c970c9cfa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad26f48b0cf4eb9a2bc22c970c9cfa3, State: Initialized, Role: FOLLOWER
I20260927 15:11:48.042078 29244 consensus_queue.cc:260] T 00000000000000000000000000000000 P aad26f48b0cf4eb9a2bc22c970c9cfa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad26f48b0cf4eb9a2bc22c970c9cfa3" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38395 } }
I20260927 15:11:48.042217 29244 raft_consensus.cc:388] T 00000000000000000000000000000000 P aad26f48b0cf4eb9a2bc22c970c9cfa3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:48.042323 29244 raft_consensus.cc:482] T 00000000000000000000000000000000 P aad26f48b0cf4eb9a2bc22c970c9cfa3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:48.042423 29244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aad26f48b0cf4eb9a2bc22c970c9cfa3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:48.043596 29244 raft_consensus.cc:504] T 00000000000000000000000000000000 P aad26f48b0cf4eb9a2bc22c970c9cfa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad26f48b0cf4eb9a2bc22c970c9cfa3" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38395 } }
I20260927 15:11:48.043972 29244 leader_election.cc:302] T 00000000000000000000000000000000 P aad26f48b0cf4eb9a2bc22c970c9cfa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aad26f48b0cf4eb9a2bc22c970c9cfa3; no voters:
I20260927 15:11:48.044332 29244 leader_election.cc:288] T 00000000000000000000000000000000 P aad26f48b0cf4eb9a2bc22c970c9cfa3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:48.044342 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:48.044449 29249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aad26f48b0cf4eb9a2bc22c970c9cfa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:48.044673 29249 raft_consensus.cc:686] T 00000000000000000000000000000000 P aad26f48b0cf4eb9a2bc22c970c9cfa3 [term 1 LEADER]: Becoming Leader. State: Replica: aad26f48b0cf4eb9a2bc22c970c9cfa3, State: Running, Role: LEADER
I20260927 15:11:48.045102 29249 consensus_queue.cc:237] T 00000000000000000000000000000000 P aad26f48b0cf4eb9a2bc22c970c9cfa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad26f48b0cf4eb9a2bc22c970c9cfa3" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38395 } }
I20260927 15:11:48.045656 29244 sys_catalog.cc:564] T 00000000000000000000000000000000 P aad26f48b0cf4eb9a2bc22c970c9cfa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:48.046871 29250 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad26f48b0cf4eb9a2bc22c970c9cfa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aad26f48b0cf4eb9a2bc22c970c9cfa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad26f48b0cf4eb9a2bc22c970c9cfa3" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38395 } } }
I20260927 15:11:48.046999 29250 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad26f48b0cf4eb9a2bc22c970c9cfa3 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:48.047350 29251 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad26f48b0cf4eb9a2bc22c970c9cfa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aad26f48b0cf4eb9a2bc22c970c9cfa3. Latest consensus state: current_term: 1 leader_uuid: "aad26f48b0cf4eb9a2bc22c970c9cfa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad26f48b0cf4eb9a2bc22c970c9cfa3" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 38395 } } }
I20260927 15:11:48.047415 29264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:48.047436 29251 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad26f48b0cf4eb9a2bc22c970c9cfa3 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:48.051121 29264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:48.056255 29247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:48.046619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60781 (local address 127.7.17.254:38395)
0927 15:11:48.046754 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:48.046759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:48.048044 (+ 1285us) server_negotiation.cc:407] Connection header received
0927 15:11:48.048199 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:48.048202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:48.048257 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:48.048345 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:48.049295 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:48.050060 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:48.051159 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:48.051508 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:48.051617 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:48.053256 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:48.053260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:48.053263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:48.053290 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:48.054579 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:48.054976 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:48.054980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:48.054982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:48.055058 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:48.055309 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:48.055313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:48.055315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:48.055543 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:48.055661 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:48.055978 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 15:11:48.056093 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:11:48.062717 29264 catalog_manager.cc:1347] Generated new cluster ID: 2329cbad7bbc44a1a0d83f27bfc6f273
I20260927 15:11:48.062809 29264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:48.075206 29264 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:48.076558 29264 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:48.085680 29264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aad26f48b0cf4eb9a2bc22c970c9cfa3: Generated new TSK 0
I20260927 15:11:48.086508 29264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:48.094350 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 29176
2026-09-27T15:11:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:48 dist-test-slave-hhpq krb5kdc[29275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:48 dist-test-slave-hhpq krb5kdc[29275](info): set up 2 sockets
Sep 27 15:11:48 dist-test-slave-hhpq krb5kdc[29275](info): commencing operation
krb5kdc: starting...
W20260927 15:11:49.044955 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:11:49 dist-test-slave-hhpq krb5kdc[29275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:11:49.165930 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:42247
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:42247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:49.277331 29288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:49.277642 29288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:49.277707 29288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:49.281802 29288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:49.281880 29288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:49.281908 29288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:49.281929 29288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:49.281955 29288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:49.287422 29288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:42247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:42247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:49.288606 29288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:49.289814 29288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:49.298931 29294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:49.298933 29296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:49.298933 29293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:49.299103 29288 server_base.cc:1034] running on GCE node
I20260927 15:11:49.299914 29288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:49.301222 29288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:49.302429 29288 hybrid_clock.cc:648] HybridClock initialized: now 1790521909302401 us; error 49 us; skew 500 ppm
Sep 27 15:11:49 dist-test-slave-hhpq krb5kdc[29275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521909, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:49.305399 29288 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:11:49.306331 29288 webserver.cc:460] Webserver started at http://127.7.17.254:34301/ using document root <none> and password file <none>
I20260927 15:11:49.306895 29288 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:49.306959 29288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:49.307148 29288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:49.309260 29288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "6d9f4f8d32cf4772a316749ba5ec44d9"
format_stamp: "Formatted at 2026-09-27 15:11:49 on dist-test-slave-hhpq"
server_key: "b54df93fa65d14a0a59539d2c5f4bfec"
server_key_iv: "4ad1899357836289c00d8d863dc1f9ff"
server_key_version: "encryptionkey@0"
I20260927 15:11:49.309819 29288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "6d9f4f8d32cf4772a316749ba5ec44d9"
format_stamp: "Formatted at 2026-09-27 15:11:49 on dist-test-slave-hhpq"
server_key: "b54df93fa65d14a0a59539d2c5f4bfec"
server_key_iv: "4ad1899357836289c00d8d863dc1f9ff"
server_key_version: "encryptionkey@0"
I20260927 15:11:49.313740 29288 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:11:49.316319 29303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:49.317692 29288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:49.317821 29288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "6d9f4f8d32cf4772a316749ba5ec44d9"
format_stamp: "Formatted at 2026-09-27 15:11:49 on dist-test-slave-hhpq"
server_key: "b54df93fa65d14a0a59539d2c5f4bfec"
server_key_iv: "4ad1899357836289c00d8d863dc1f9ff"
server_key_version: "encryptionkey@0"
I20260927 15:11:49.317943 29288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:49.343472 29288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:49.346877 29288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:49.347086 29288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:49.354064 29288 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:42247
I20260927 15:11:49.354089 29355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:42247 every 8 connection(s)
I20260927 15:11:49.355087 29288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:49.357935 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:49.362707 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 29288
I20260927 15:11:49.362816 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:49.363060 7239 external_mini_cluster.cc:1428] Setting key 9f67d3158c773e8a8fbf13f8efde95c6
I20260927 15:11:49.363783 29356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d9f4f8d32cf4772a316749ba5ec44d9: Bootstrap starting.
I20260927 15:11:49.366497 29356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d9f4f8d32cf4772a316749ba5ec44d9: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:49.367451 29356 log.cc:824] T 00000000000000000000000000000000 P 6d9f4f8d32cf4772a316749ba5ec44d9: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:49.369659 29356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d9f4f8d32cf4772a316749ba5ec44d9: No bootstrap required, opened a new log
I20260927 15:11:49.373764 29359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:49.364554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52479 (local address 127.7.17.254:42247)
0927 15:11:49.365058 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:49.365078 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:49.365125 (+ 47us) server_negotiation.cc:407] Connection header received
0927 15:11:49.365998 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:49.366024 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:49.366497 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:49.366927 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:49.367366 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:49.368684 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:49.369677 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:49.370367 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:49.370536 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:49.371375 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:49.371382 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:49.371401 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:49.371968 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:49.372054 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:49.372087 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:49.372190 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:49.372336 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 15:11:49.372618 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":145,"threads_started":1}
I20260927 15:11:49.374312 29356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d9f4f8d32cf4772a316749ba5ec44d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d9f4f8d32cf4772a316749ba5ec44d9" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42247 } }
I20260927 15:11:49.374571 29356 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d9f4f8d32cf4772a316749ba5ec44d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:49.374630 29356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d9f4f8d32cf4772a316749ba5ec44d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d9f4f8d32cf4772a316749ba5ec44d9, State: Initialized, Role: FOLLOWER
I20260927 15:11:49.375125 29356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d9f4f8d32cf4772a316749ba5ec44d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d9f4f8d32cf4772a316749ba5ec44d9" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42247 } }
I20260927 15:11:49.375294 29356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d9f4f8d32cf4772a316749ba5ec44d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:49.375355 29356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d9f4f8d32cf4772a316749ba5ec44d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:49.375447 29356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d9f4f8d32cf4772a316749ba5ec44d9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:49.376573 29356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d9f4f8d32cf4772a316749ba5ec44d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d9f4f8d32cf4772a316749ba5ec44d9" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42247 } }
I20260927 15:11:49.376981 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:49.376989 29356 leader_election.cc:302] T 00000000000000000000000000000000 P 6d9f4f8d32cf4772a316749ba5ec44d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d9f4f8d32cf4772a316749ba5ec44d9; no voters:
I20260927 15:11:49.377342 29356 leader_election.cc:288] T 00000000000000000000000000000000 P 6d9f4f8d32cf4772a316749ba5ec44d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:49.377511 29361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d9f4f8d32cf4772a316749ba5ec44d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:49.377723 29361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d9f4f8d32cf4772a316749ba5ec44d9 [term 1 LEADER]: Becoming Leader. State: Replica: 6d9f4f8d32cf4772a316749ba5ec44d9, State: Running, Role: LEADER
I20260927 15:11:49.378057 29361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d9f4f8d32cf4772a316749ba5ec44d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d9f4f8d32cf4772a316749ba5ec44d9" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42247 } }
I20260927 15:11:49.378729 29356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d9f4f8d32cf4772a316749ba5ec44d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:49.379911 29363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d9f4f8d32cf4772a316749ba5ec44d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d9f4f8d32cf4772a316749ba5ec44d9. Latest consensus state: current_term: 1 leader_uuid: "6d9f4f8d32cf4772a316749ba5ec44d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d9f4f8d32cf4772a316749ba5ec44d9" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42247 } } }
I20260927 15:11:49.380021 29363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d9f4f8d32cf4772a316749ba5ec44d9 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:49.380429 29375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:49.380477 29362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d9f4f8d32cf4772a316749ba5ec44d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d9f4f8d32cf4772a316749ba5ec44d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d9f4f8d32cf4772a316749ba5ec44d9" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 42247 } } }
I20260927 15:11:49.380573 29362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d9f4f8d32cf4772a316749ba5ec44d9 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:49.383832 29375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:49.384090 29359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:49.380326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38451 (local address 127.7.17.254:42247)
0927 15:11:49.380435 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:49.380440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:49.380466 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:11:49.380505 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:49.380533 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:49.380582 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:49.380664 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:49.381092 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:49.381820 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:49.382616 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:49.382828 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:49.382912 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:49.383694 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:49.383698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:49.383700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:49.383713 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:49.383739 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:49.383750 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:49.383810 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:49.383908 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 15:11:49.383960 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 15:11:49.389802 29375 catalog_manager.cc:1347] Generated new cluster ID: 9f67a6498d124c898261fc7aa3b6376d
I20260927 15:11:49.389936 29375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:49.413941 29375 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:49.415381 29375 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:49.424757 29375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d9f4f8d32cf4772a316749ba5ec44d9: Generated new TSK 0
I20260927 15:11:49.425724 29375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:49.429971 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 29288
2026-09-27T15:11:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:49 dist-test-slave-hhpq krb5kdc[29387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:49 dist-test-slave-hhpq krb5kdc[29387](info): set up 2 sockets
Sep 27 15:11:49 dist-test-slave-hhpq krb5kdc[29387](info): commencing operation
krb5kdc: starting...
W20260927 15:11:50.388904 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:11:50 dist-test-slave-hhpq krb5kdc[29387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:11:50.512233 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:36561
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:36725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:36561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:50.624454 29400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:50.624786 29400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:50.624881 29400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:50.629155 29400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:50.629240 29400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:50.629267 29400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:50.629288 29400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:50.629309 29400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:50.635033 29400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:36725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:36561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:36561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:50.636246 29400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:50.637360 29400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:50.646468 29408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:50.646466 29405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:50.646571 29400 server_base.cc:1034] running on GCE node
W20260927 15:11:50.646459 29406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:50.647468 29400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:50.648800 29400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:50.650048 29400 hybrid_clock.cc:648] HybridClock initialized: now 1790521910650016 us; error 48 us; skew 500 ppm
Sep 27 15:11:50 dist-test-slave-hhpq krb5kdc[29387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521910, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:50.653316 29400 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:11:50.654425 29400 webserver.cc:460] Webserver started at http://127.7.17.254:39417/ using document root <none> and password file <none>
I20260927 15:11:50.655040 29400 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:50.655107 29400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:50.655295 29400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:50.657629 29400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "33baac5a4fac4a4499f70f4135aa1ef7"
format_stamp: "Formatted at 2026-09-27 15:11:50 on dist-test-slave-hhpq"
server_key: "61a6847dacda2e1d1885758b49f4b0e7"
server_key_iv: "dab4708598a2aee1defbc0498fe8a450"
server_key_version: "encryptionkey@0"
I20260927 15:11:50.658257 29400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "33baac5a4fac4a4499f70f4135aa1ef7"
format_stamp: "Formatted at 2026-09-27 15:11:50 on dist-test-slave-hhpq"
server_key: "61a6847dacda2e1d1885758b49f4b0e7"
server_key_iv: "dab4708598a2aee1defbc0498fe8a450"
server_key_version: "encryptionkey@0"
I20260927 15:11:50.662467 29400 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:11:50.665391 29415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:50.666901 29400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:50.667047 29400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "33baac5a4fac4a4499f70f4135aa1ef7"
format_stamp: "Formatted at 2026-09-27 15:11:50 on dist-test-slave-hhpq"
server_key: "61a6847dacda2e1d1885758b49f4b0e7"
server_key_iv: "dab4708598a2aee1defbc0498fe8a450"
server_key_version: "encryptionkey@0"
I20260927 15:11:50.667167 29400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:50.672147 29400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:50.675446 29400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:50.675606 29400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:50.682842 29400 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:36561
I20260927 15:11:50.682866 29467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:36561 every 8 connection(s)
I20260927 15:11:50.683979 29400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:50.687040 29468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:50.688773 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 29400
I20260927 15:11:50.688926 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:50.689191 7239 external_mini_cluster.cc:1428] Setting key 4b8cae5786f0043732af5fa163de9acd
I20260927 15:11:50.694096 29468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33baac5a4fac4a4499f70f4135aa1ef7: Bootstrap starting.
I20260927 15:11:50.695156 29471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:50.690701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57121 (local address 127.7.17.254:36561)
0927 15:11:50.691211 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:50.691227 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:50.691271 (+ 44us) server_negotiation.cc:407] Connection header received
0927 15:11:50.692069 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:50.692093 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:50.692447 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:50.692866 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:50.693786 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:50.693792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:50.693808 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:50.693820 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:50.693882 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:50.693904 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:50.694049 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:50.694128 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 15:11:50.694329 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":143,"threads_started":1}
I20260927 15:11:50.696628 29468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33baac5a4fac4a4499f70f4135aa1ef7: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:50.697503 29468 log.cc:824] T 00000000000000000000000000000000 P 33baac5a4fac4a4499f70f4135aa1ef7: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:50.698105 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:50.699589 29468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33baac5a4fac4a4499f70f4135aa1ef7: No bootstrap required, opened a new log
I20260927 15:11:50.701768 29471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:50.700115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57205 (local address 127.7.17.254:36561)
0927 15:11:50.700223 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:50.700227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:50.700415 (+ 188us) server_negotiation.cc:407] Connection header received
0927 15:11:50.700530 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:50.700533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:50.700568 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:50.700630 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:50.701413 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:50.701417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:50.701420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:11:50.701432 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:50.701459 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:11:50.701469 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:50.701540 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:50.701610 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 15:11:50.701651 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 15:11:50.703462 29468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33baac5a4fac4a4499f70f4135aa1ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33baac5a4fac4a4499f70f4135aa1ef7" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36561 } }
I20260927 15:11:50.703719 29468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33baac5a4fac4a4499f70f4135aa1ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:11:50.703765 29468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33baac5a4fac4a4499f70f4135aa1ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33baac5a4fac4a4499f70f4135aa1ef7, State: Initialized, Role: FOLLOWER
I20260927 15:11:50.704211 29468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33baac5a4fac4a4499f70f4135aa1ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33baac5a4fac4a4499f70f4135aa1ef7" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36561 } }
I20260927 15:11:50.704347 29468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33baac5a4fac4a4499f70f4135aa1ef7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:11:50.704397 29468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33baac5a4fac4a4499f70f4135aa1ef7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:11:50.704473 29468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33baac5a4fac4a4499f70f4135aa1ef7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:50.705495 29468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33baac5a4fac4a4499f70f4135aa1ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33baac5a4fac4a4499f70f4135aa1ef7" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36561 } }
I20260927 15:11:50.705920 29468 leader_election.cc:302] T 00000000000000000000000000000000 P 33baac5a4fac4a4499f70f4135aa1ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33baac5a4fac4a4499f70f4135aa1ef7; no voters:
I20260927 15:11:50.706255 29468 leader_election.cc:288] T 00000000000000000000000000000000 P 33baac5a4fac4a4499f70f4135aa1ef7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:11:50.706411 29478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33baac5a4fac4a4499f70f4135aa1ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:50.706624 29478 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33baac5a4fac4a4499f70f4135aa1ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 33baac5a4fac4a4499f70f4135aa1ef7, State: Running, Role: LEADER
I20260927 15:11:50.707037 29478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33baac5a4fac4a4499f70f4135aa1ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33baac5a4fac4a4499f70f4135aa1ef7" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36561 } }
I20260927 15:11:50.707574 29468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33baac5a4fac4a4499f70f4135aa1ef7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:50.708691 29479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33baac5a4fac4a4499f70f4135aa1ef7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33baac5a4fac4a4499f70f4135aa1ef7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33baac5a4fac4a4499f70f4135aa1ef7" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36561 } } }
I20260927 15:11:50.708846 29479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33baac5a4fac4a4499f70f4135aa1ef7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:50.709237 29487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:50.709205 29480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33baac5a4fac4a4499f70f4135aa1ef7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33baac5a4fac4a4499f70f4135aa1ef7. Latest consensus state: current_term: 1 leader_uuid: "33baac5a4fac4a4499f70f4135aa1ef7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33baac5a4fac4a4499f70f4135aa1ef7" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36561 } } }
I20260927 15:11:50.709394 29480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33baac5a4fac4a4499f70f4135aa1ef7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:50.712368 29487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:50.719022 29487 catalog_manager.cc:1347] Generated new cluster ID: 40ca9d77281944f5b2e0546bfec5bc8e
I20260927 15:11:50.719115 29487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:50.731913 29487 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:50.732350 29487 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:50.743064 29487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33baac5a4fac4a4499f70f4135aa1ef7: Generated new TSK 0
I20260927 15:11:50.743816 29487 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:11:50.756224 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 29400
2026-09-27T15:11:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1326 ms)
[----------] 18 tests from AuthTokenIssuingTest (23928 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:50 dist-test-slave-hhpq krb5kdc[29499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:50 dist-test-slave-hhpq krb5kdc[29499](info): set up 2 sockets
Sep 27 15:11:50 dist-test-slave-hhpq krb5kdc[29499](info): commencing operation
krb5kdc: starting...
W20260927 15:11:51.709026 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:11:51 dist-test-slave-hhpq krb5kdc[29499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:11:51.836001 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:35127
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:43521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:35127,127.7.17.253:41365,127.7.17.252:34181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:51.949581 29512 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:11:51.949911 29512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:51.949935 29512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:51.949980 29512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:51.954159 29512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:51.954234 29512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:51.954253 29512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:51.954268 29512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:51.954282 29512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:51.959818 29512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:43521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:35127,127.7.17.253:41365,127.7.17.252:34181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:35127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:51.960996 29512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:51.962054 29512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:51.970711 29517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:51.970783 29512 server_base.cc:1034] running on GCE node
W20260927 15:11:51.970705 29520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:51.970705 29518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:51.971629 29512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:51.972901 29512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:51.974109 29512 hybrid_clock.cc:648] HybridClock initialized: now 1790521911974083 us; error 47 us; skew 500 ppm
Sep 27 15:11:51 dist-test-slave-hhpq krb5kdc[29499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521911, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:51.977203 29512 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:11:51.978180 29512 webserver.cc:460] Webserver started at http://127.7.17.254:34563/ using document root <none> and password file <none>
I20260927 15:11:51.978801 29512 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:51.978886 29512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:51.979100 29512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:51.981352 29512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "b3f939d9290c4b258bb8ebbfad06ef12"
format_stamp: "Formatted at 2026-09-27 15:11:51 on dist-test-slave-hhpq"
server_key: "6c05135dbc3d316b85c91ff28980615c"
server_key_iv: "3f1204962b4a7ffdf8459dfba2dae1fa"
server_key_version: "encryptionkey@0"
I20260927 15:11:51.981962 29512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "b3f939d9290c4b258bb8ebbfad06ef12"
format_stamp: "Formatted at 2026-09-27 15:11:51 on dist-test-slave-hhpq"
server_key: "6c05135dbc3d316b85c91ff28980615c"
server_key_iv: "3f1204962b4a7ffdf8459dfba2dae1fa"
server_key_version: "encryptionkey@0"
I20260927 15:11:51.985994 29512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:11:51.988623 29527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:51.990016 29512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:51.990159 29512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "b3f939d9290c4b258bb8ebbfad06ef12"
format_stamp: "Formatted at 2026-09-27 15:11:51 on dist-test-slave-hhpq"
server_key: "6c05135dbc3d316b85c91ff28980615c"
server_key_iv: "3f1204962b4a7ffdf8459dfba2dae1fa"
server_key_version: "encryptionkey@0"
I20260927 15:11:51.990286 29512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:52.002691 29512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:52.005996 29512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:52.006189 29512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:52.013221 29512 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:35127
I20260927 15:11:52.013496 29579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:35127 every 8 connection(s)
I20260927 15:11:52.014262 29512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:52.017208 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:52.021085 29580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35127 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:11:52.022599 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 29512
I20260927 15:11:52.022712 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:52.023001 7239 external_mini_cluster.cc:1428] Setting key 462f397796171b41afe335d8a3aa4b76
WARNING: no policy specified for kudu/127.7.17.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.253@KRBTEST.COM" created.
Sep 27 15:11:52 dist-test-slave-hhpq krb5kdc[29499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521911, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:11:52.039382 29584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.024384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42308 (local address 127.7.17.254:35127)
0927 15:11:52.024693 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.024704 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.024748 (+ 44us) server_negotiation.cc:407] Connection header received
0927 15:11:52.025576 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.025580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.025775 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.025954 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.026896 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.027775 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.028778 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.029215 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.029368 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.034136 (+ 4768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:52.034145 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:52.034163 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:52.034196 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:52.035892 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.036568 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.036574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.036576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.036707 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.037157 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.037160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.037161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.037630 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:52.037750 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:52.038280 (+ 530us) server_negotiation.cc:299] Negotiation successful
0927 15:11:52.038544 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":215,"thread_start_us":102,"threads_started":1}
I20260927 15:11:52.039386 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.023534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:35127 (local address 127.0.0.1:42308)
0927 15:11:52.024117 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.024183 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:52.025244 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:52.025960 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:52.025966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:52.026422 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:52.026742 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:52.026759 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.027903 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.027905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:52.028665 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:52.028674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.029374 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.029378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:52.029475 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.030053 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:52.030072 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:52.033908 (+ 3836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:52.036044 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:52.036057 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:52.036072 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:52.036439 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:52.036806 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:52.036809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:52.036812 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:52.037016 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:52.037763 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:52.037771 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:52.037986 (+ 215us) client_negotiation.cc:769] Sending connection context
0927 15:11:52.038197 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 15:11:52.038621 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":79,"thread_start_us":142,"threads_started":1}
I20260927 15:11:52.042083 29580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:11:52.042841 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.042571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:41365 (local address 127.0.0.1:56624)
0927 15:11:52.042686 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.042756 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.253:41365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 15:11:52.043087 29530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.17.253:41365: connect: Connection refused (error 111)
I20260927 15:11:52.043895 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.043695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:41365 (local address 127.0.0.1:56626)
0927 15:11:52.043799 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.043842 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.253:41365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 15:11:52.044034 29580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.253:41365: Network error: Client connection negotiation failed: client connection to 127.7.17.253:41365: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.253.keytab.
Entry for principal kudu/127.7.17.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.253.keytab.
I20260927 15:11:52.048403 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.253:41365
--webserver_interface=127.7.17.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.253.keytab
--principal=kudu/127.7.17.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:43521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:35127,127.7.17.253:41365,127.7.17.252:34181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:11:52.093266 29580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } attempt: 1
I20260927 15:11:52.094197 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.093857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:41365 (local address 127.0.0.1:56642)
0927 15:11:52.094049 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.094118 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.253:41365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 15:11:52.094911 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.094682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:41365 (local address 127.0.0.1:56646)
0927 15:11:52.094789 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.094838 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.253:41365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 15:11:52.095029 29580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.253:41365: Network error: Client connection negotiation failed: client connection to 127.7.17.253:41365: connect: Connection refused (error 111)
W20260927 15:11:52.163040 29587 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260927 15:11:52.163223 29580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } attempt: 2
W20260927 15:11:52.163380 29587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:52.163405 29587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:52.163457 29587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 15:11:52.164077 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.163731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:41365 (local address 127.0.0.1:56648)
0927 15:11:52.163905 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.163982 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.253:41365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 15:11:52.164927 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.164672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:41365 (local address 127.0.0.1:56654)
0927 15:11:52.164773 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.164853 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.253:41365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 15:11:52.165074 29580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.253:41365: Network error: Client connection negotiation failed: client connection to 127.7.17.253:41365: connect: Connection refused (error 111)
W20260927 15:11:52.168079 29587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:52.168138 29587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:52.168159 29587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:52.168174 29587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:52.168188 29587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:52.173848 29587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:43521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:35127,127.7.17.253:41365,127.7.17.252:34181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.253.keytab
--principal=kudu/127.7.17.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.253:41365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.17.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:52.174981 29587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:52.176028 29587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:52.184489 29593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:52.184640 29587 server_base.cc:1034] running on GCE node
W20260927 15:11:52.184512 29595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:52.184492 29592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:52.185387 29587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:52.186604 29587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:52.187783 29587 hybrid_clock.cc:648] HybridClock initialized: now 1790521912187762 us; error 35 us; skew 500 ppm
Sep 27 15:11:52 dist-test-slave-hhpq krb5kdc[29499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521912, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:52.190828 29587 init.cc:377] Logged in from keytab as kudu/127.7.17.253@KRBTEST.COM (short username kudu)
I20260927 15:11:52.191831 29587 webserver.cc:460] Webserver started at http://127.7.17.253:45831/ using document root <none> and password file <none>
I20260927 15:11:52.192407 29587 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:52.192481 29587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:52.192685 29587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:52.194944 29587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-1/data/instance:
uuid: "db81012d9e3843dab6f157cdfe445e29"
format_stamp: "Formatted at 2026-09-27 15:11:52 on dist-test-slave-hhpq"
server_key: "e930306734160e68ad2fa9ff62508f85"
server_key_iv: "c60ce2a6ffd3ff4eb3ce9855e6a997fd"
server_key_version: "encryptionkey@0"
I20260927 15:11:52.195505 29587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-1/wal/instance:
uuid: "db81012d9e3843dab6f157cdfe445e29"
format_stamp: "Formatted at 2026-09-27 15:11:52 on dist-test-slave-hhpq"
server_key: "e930306734160e68ad2fa9ff62508f85"
server_key_iv: "c60ce2a6ffd3ff4eb3ce9855e6a997fd"
server_key_version: "encryptionkey@0"
I20260927 15:11:52.199254 29587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:11:52.201709 29602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:52.202996 29587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:11:52.203114 29587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-1/wal
uuid: "db81012d9e3843dab6f157cdfe445e29"
format_stamp: "Formatted at 2026-09-27 15:11:52 on dist-test-slave-hhpq"
server_key: "e930306734160e68ad2fa9ff62508f85"
server_key_iv: "c60ce2a6ffd3ff4eb3ce9855e6a997fd"
server_key_version: "encryptionkey@0"
I20260927 15:11:52.203209 29587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:52.238210 29587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:52.241080 29587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:52.241259 29587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:52.248005 29587 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.253:41365
I20260927 15:11:52.248267 29654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.253:41365 every 8 connection(s)
I20260927 15:11:52.249099 29587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-1/data/info.pb
I20260927 15:11:52.251916 29655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:52.255066 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 29587
I20260927 15:11:52.255209 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-1/wal/instance
I20260927 15:11:52.255476 7239 external_mini_cluster.cc:1428] Setting key c31a1a4d1e3c2442870583d5487aa5af
I20260927 15:11:52.256261 29580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } attempt: 3
I20260927 15:11:52.256107 29655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35127 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.17.252@KRBTEST.COM; defaulting to no policy
Sep 27 15:11:52 dist-test-slave-hhpq krb5kdc[29499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521911, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for kudu/127.7.17.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.252@KRBTEST.COM" created.
Sep 27 15:11:52 dist-test-slave-hhpq krb5kdc[29499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521912, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.253@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:11:52.268679 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.256779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:41365 (local address 127.0.0.1:56662)
0927 15:11:52.256992 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.257011 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:52.257122 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:52.259625 (+ 2503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:52.259628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:52.259815 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:52.259953 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:52.259961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.261445 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.261449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:52.262259 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:52.262271 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.263100 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.263104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:52.263197 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.263677 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:52.263692 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:52.264950 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:52.266720 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:52.266723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:52.266725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:52.266918 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:52.267315 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:52.267318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:52.267321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:52.267427 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:52.268132 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:52.268136 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:52.268255 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 15:11:52.268346 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 15:11:52.268498 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"spinlock_wait_cycles":4224}
I20260927 15:11:52.269806 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.257771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56662 (local address 127.7.17.253:41365)
0927 15:11:52.258216 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.258231 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.258266 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:11:52.259150 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.259175 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.259428 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.259655 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.260097 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.261319 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.262372 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.262978 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.263386 (+ 408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.265085 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:52.265091 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:52.265103 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:52.265124 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:52.266599 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.267017 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.267022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.267028 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.267175 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.267534 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.267538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.267540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.268026 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:52.268150 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:52.268529 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 15:11:52.268871 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"spinlock_wait_cycles":6912,"thread_start_us":122,"threads_started":1}
I20260927 15:11:52.271611 29659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.258190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:35127 (local address 127.0.0.1:42318)
0927 15:11:52.258577 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.258624 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:52.259152 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:52.259532 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:52.259542 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:52.259981 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:52.260261 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:52.260271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.261171 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.261175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:52.262007 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:52.262015 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.263129 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.263133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:52.263321 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.263845 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:52.263867 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:52.268046 (+ 4179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:52.270001 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:52.270008 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:52.270014 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:52.270277 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:52.270546 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:52.270548 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:52.270550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:52.270636 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:52.271010 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:52.271016 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:52.271147 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 15:11:52.271297 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 15:11:52.271471 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"spinlock_wait_cycles":2432,"thread_start_us":129,"threads_started":1}
I20260927 15:11:52.271641 29584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.258232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42318 (local address 127.7.17.254:35127)
0927 15:11:52.258360 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.258365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.258709 (+ 344us) server_negotiation.cc:407] Connection header received
0927 15:11:52.259340 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.259343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.259395 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.259467 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.260385 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.261054 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.262120 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.262953 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.263065 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.268201 (+ 5136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:52.268205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:52.268208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:52.268235 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:52.269883 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.270388 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.270391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.270393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.270469 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.270710 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.270712 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.270713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.270869 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:52.271005 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:52.271377 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 15:11:52.271501 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:11:52.272230 29580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:11:52.272959 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.272692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:34181 (local address 127.0.0.1:59614)
0927 15:11:52.272809 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.272888 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:34181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 15:11:52.273042 29655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:11:52.273905 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.273533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:34181 (local address 127.0.0.1:59620)
0927 15:11:52.273784 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.273832 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:34181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194}
W20260927 15:11:52.274092 29580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.252:34181: Network error: Client connection negotiation failed: client connection to 127.7.17.252:34181: connect: Connection refused (error 111)
Sep 27 15:11:52 dist-test-slave-hhpq krb5kdc[29499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521912, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.253@KRBTEST.COM for kudu/127.7.17.253@KRBTEST.COM
I20260927 15:11:52.281036 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.273674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56670 (local address 127.7.17.253:41365)
0927 15:11:52.273773 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.273776 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.273789 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:11:52.273821 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.273824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.273864 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.273942 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.274483 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.274984 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.276033 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.276255 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.276321 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.278318 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:52.278321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:52.278323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:52.278344 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:52.279577 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.279949 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.279953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.279954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.280018 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.280237 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.280240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.280242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.280410 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:52.280487 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:52.280698 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 15:11:52.280873 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":35}
I20260927 15:11:52.281036 29659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.273525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:41365 (local address 127.0.0.1:56670)
0927 15:11:52.273624 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.273636 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:52.273714 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:52.273966 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:52.273969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:52.274247 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:52.274373 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:52.274381 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.275084 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.275087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:52.275934 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:52.275945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.276351 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.276354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:52.276453 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.276905 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:52.276920 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:52.278186 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:52.279672 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:52.279675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:52.279677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:52.279853 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:52.280091 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:52.280094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:52.280096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:52.280168 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:52.280489 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:52.280492 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:52.280570 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 15:11:52.280649 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 15:11:52.280792 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 15:11:52.281795 29655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:11:52.282435 29659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.282188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:34181 (local address 127.0.0.1:59622)
0927 15:11:52.282295 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.282377 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:34181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 15:11:52.282620 29606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.17.252:34181: connect: Connection refused (error 111)
I20260927 15:11:52.283408 29659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.283213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:34181 (local address 127.0.0.1:59636)
0927 15:11:52.283314 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.283356 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:34181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 15:11:52.283524 29655 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.252:34181: Network error: Client connection negotiation failed: client connection to 127.7.17.252:34181: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.252.keytab.
Entry for principal kudu/127.7.17.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.252.keytab.
I20260927 15:11:52.286266 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.252:34181
--webserver_interface=127.7.17.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.252.keytab
--principal=kudu/127.7.17.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:43521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:35127,127.7.17.253:41365,127.7.17.252:34181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:11:52.305260 29580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } attempt: 1
I20260927 15:11:52.306046 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.305765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:34181 (local address 127.0.0.1:59648)
0927 15:11:52.305908 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.305967 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:34181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 15:11:52.306895 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.306593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:34181 (local address 127.0.0.1:59660)
0927 15:11:52.306768 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.306819 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:34181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260927 15:11:52.307018 29580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.252:34181: Network error: Client connection negotiation failed: client connection to 127.7.17.252:34181: connect: Connection refused (error 111)
I20260927 15:11:52.332798 29655 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } attempt: 1
I20260927 15:11:52.333647 29659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.333377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:34181 (local address 127.0.0.1:59666)
0927 15:11:52.333512 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.333571 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:34181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 15:11:52.334375 29659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.334169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:34181 (local address 127.0.0.1:59674)
0927 15:11:52.334272 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.334316 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:34181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 15:11:52.334479 29655 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.252:34181: Network error: Client connection negotiation failed: client connection to 127.7.17.252:34181: connect: Connection refused (error 111)
I20260927 15:11:52.382171 29580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } attempt: 2
I20260927 15:11:52.383006 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.382707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:34181 (local address 127.0.0.1:59680)
0927 15:11:52.382863 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.382928 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:34181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 15:11:52.383669 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.383469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:34181 (local address 127.0.0.1:59688)
0927 15:11:52.383565 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.383609 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:34181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 15:11:52.383781 29580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.252:34181: Network error: Client connection negotiation failed: client connection to 127.7.17.252:34181: connect: Connection refused (error 111)
W20260927 15:11:52.399204 29662 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:11:52.399514 29662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:52.399561 29662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:52.399626 29662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 15:11:52.402645 29655 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } attempt: 2
I20260927 15:11:52.403415 29659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.403128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:34181 (local address 127.0.0.1:59704)
0927 15:11:52.403265 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.403323 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:34181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 15:11:52.403728 29662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:52.403797 29662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:52.403825 29662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:52.403920 29662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:52.403952 29662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:52.404256 29659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.404009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:34181 (local address 127.0.0.1:59720)
0927 15:11:52.404126 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.404184 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:34181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 15:11:52.404399 29655 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.252:34181: Network error: Client connection negotiation failed: client connection to 127.7.17.252:34181: connect: Connection refused (error 111)
I20260927 15:11:52.409747 29662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:43521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:35127,127.7.17.253:41365,127.7.17.252:34181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.252.keytab
--principal=kudu/127.7.17.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.252:34181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.17.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:52.411219 29662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:52.412667 29662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:52.421808 29668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:52.421919 29667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:52.421808 29670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:52.422116 29662 server_base.cc:1034] running on GCE node
I20260927 15:11:52.422674 29662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:52.423803 29662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:52.424955 29662 hybrid_clock.cc:648] HybridClock initialized: now 1790521912424931 us; error 37 us; skew 500 ppm
Sep 27 15:11:52 dist-test-slave-hhpq krb5kdc[29499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521912, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:52.427549 29662 init.cc:377] Logged in from keytab as kudu/127.7.17.252@KRBTEST.COM (short username kudu)
I20260927 15:11:52.428484 29662 webserver.cc:460] Webserver started at http://127.7.17.252:44927/ using document root <none> and password file <none>
I20260927 15:11:52.429072 29662 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:52.429152 29662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:52.429354 29662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:52.431422 29662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-2/data/instance:
uuid: "84479b3c32174b28a672e4d9f8401ce0"
format_stamp: "Formatted at 2026-09-27 15:11:52 on dist-test-slave-hhpq"
server_key: "787ba11a626ff62f5e2afd4c581a3b5a"
server_key_iv: "9399924032e98336b0188a6c69a3ca71"
server_key_version: "encryptionkey@0"
I20260927 15:11:52.431982 29662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-2/wal/instance:
uuid: "84479b3c32174b28a672e4d9f8401ce0"
format_stamp: "Formatted at 2026-09-27 15:11:52 on dist-test-slave-hhpq"
server_key: "787ba11a626ff62f5e2afd4c581a3b5a"
server_key_iv: "9399924032e98336b0188a6c69a3ca71"
server_key_version: "encryptionkey@0"
I20260927 15:11:52.435819 29662 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:11:52.438349 29677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:52.439601 29662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:11:52.439716 29662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-2/wal
uuid: "84479b3c32174b28a672e4d9f8401ce0"
format_stamp: "Formatted at 2026-09-27 15:11:52 on dist-test-slave-hhpq"
server_key: "787ba11a626ff62f5e2afd4c581a3b5a"
server_key_iv: "9399924032e98336b0188a6c69a3ca71"
server_key_version: "encryptionkey@0"
I20260927 15:11:52.439814 29662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:52.449862 29662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:52.452886 29662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:52.453104 29662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:52.460040 29662 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.252:34181
I20260927 15:11:52.460294 29729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.252:34181 every 8 connection(s)
I20260927 15:11:52.461087 29662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-2/data/info.pb
I20260927 15:11:52.462452 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 29662
I20260927 15:11:52.462574 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521636012660-7239-0/minicluster-data/master-2/wal/instance
I20260927 15:11:52.462875 7239 external_mini_cluster.cc:1428] Setting key 52518b304845dc057400d76672301170
I20260927 15:11:52.464392 29730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:52.468580 29730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35127 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:11:52 dist-test-slave-hhpq krb5kdc[29499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.252@KRBTEST.COM
Sep 27 15:11:52 dist-test-slave-hhpq krb5kdc[29499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521912, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.252@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:11:52.477543 29733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.464218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59726 (local address 127.7.17.252:34181)
0927 15:11:52.464660 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.464676 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.464713 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:11:52.465388 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.465414 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.465749 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.466053 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.466918 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.467917 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.469108 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.469591 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.469708 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.472612 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:52.472617 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:52.472626 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:52.472653 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:52.474349 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.474853 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.474860 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.474866 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.475025 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.475308 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.475311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.475313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.475769 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:52.475963 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:52.476255 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 15:11:52.476550 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":130,"threads_started":1}
I20260927 15:11:52.479532 29734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.470444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:35127 (local address 127.0.0.1:42320)
0927 15:11:52.470870 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.470924 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:52.471040 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:52.471327 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:52.471333 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:52.471781 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:52.471941 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:52.471951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.472882 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.472885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:52.473819 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:52.473831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.474287 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.474291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:52.474402 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.474885 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:52.474899 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:52.476462 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:52.478013 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:52.478021 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:52.478027 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:52.478239 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:52.478518 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:52.478521 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:52.478523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:52.478616 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:52.478966 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:52.478971 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:52.479086 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 15:11:52.479220 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 15:11:52.479395 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":167,"threads_started":1}
I20260927 15:11:52.479554 29584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.470531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42320 (local address 127.7.17.254:35127)
0927 15:11:52.470673 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.470678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.471003 (+ 325us) server_negotiation.cc:407] Connection header received
0927 15:11:52.471126 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.471129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.471183 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.471272 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.472077 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.472744 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.473940 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.474180 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.474259 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.476593 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:52.476596 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:52.476598 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:52.476623 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:52.477915 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.478329 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.478333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.478334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.478430 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.478701 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.478703 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.478704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.478884 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:52.478953 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:52.479298 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 15:11:52.479414 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:11:52.480615 29730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:11:52.482956 29580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } attempt: 3
Sep 27 15:11:52 dist-test-slave-hhpq krb5kdc[29499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521912, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.252@KRBTEST.COM for kudu/127.7.17.253@KRBTEST.COM
Sep 27 15:11:52 dist-test-slave-hhpq krb5kdc[29499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
Sep 27 15:11:52 dist-test-slave-hhpq krb5kdc[29499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521911, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for kudu/127.7.17.252@KRBTEST.COM
I20260927 15:11:52.488413 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.481167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56686 (local address 127.7.17.253:41365)
0927 15:11:52.481300 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.481304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.481316 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:11:52.481352 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.481354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.481390 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.481452 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.481803 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.482275 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.482963 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.483274 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.483369 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.485164 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:52.485168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:52.485170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:52.485193 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:52.486734 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.487194 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.487197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.487199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.487280 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.487541 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.487544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.487546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.487748 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:52.487842 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:52.488127 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 15:11:52.488237 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:11:52.488907 29734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.481047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:41365 (local address 127.0.0.1:56686)
0927 15:11:52.481146 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.481159 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:52.481221 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:52.481474 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:52.481477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:52.481619 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:52.481704 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:52.481709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.482371 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.482375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:52.482866 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:52.482873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.483376 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.483380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:52.483465 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.483852 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:52.483864 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:52.485058 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:52.486835 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:52.486839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:52.486841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:52.487063 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:52.487351 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:52.487354 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:52.487356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:52.487438 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:52.487841 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:52.487844 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:52.487944 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 15:11:52.488610 (+ 666us) client_negotiation.cc:240] Negotiation successful
0927 15:11:52.488733 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 15:11:52.488986 29584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.480145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42324 (local address 127.7.17.254:35127)
0927 15:11:52.480243 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.480247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.480300 (+ 53us) server_negotiation.cc:407] Connection header received
0927 15:11:52.480421 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.480424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.480463 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.480524 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.481279 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.481764 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.482501 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.482712 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.482793 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.485814 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:52.485818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:52.485821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:52.485843 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:52.487219 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.487608 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.487611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.487613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.487692 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.487943 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.487947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.487948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.488172 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:52.488403 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:52.488657 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 15:11:52.488793 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:11:52.489887 29730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:11:52.493048 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.483312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:34181 (local address 127.0.0.1:59730)
0927 15:11:52.483463 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.483475 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:52.483542 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:52.483766 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:52.483769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:52.483937 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:52.484030 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:52.484037 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.484764 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.484768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:52.485586 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:52.485597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.486759 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.486763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:52.486863 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.487390 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:52.487401 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:52.488819 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:52.491565 (+ 2746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:52.491568 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:52.491570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:52.491763 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:52.492019 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:52.492022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:52.492025 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:52.492105 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:52.492450 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:52.492453 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:52.492584 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 15:11:52.492691 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 15:11:52.492865 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 15:11:52.493062 29733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.483353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59730 (local address 127.7.17.252:34181)
0927 15:11:52.483508 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.483511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.483523 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:11:52.483626 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.483628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.483667 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.483743 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.484143 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.484662 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.485700 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.486405 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.487035 (+ 630us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.489275 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:52.489279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:52.489281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:52.489306 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:52.491486 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.491850 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.491862 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.491864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.491937 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.492183 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.492186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.492188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.492362 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:52.492454 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:52.492755 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 15:11:52.492906 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
Sep 27 15:11:52 dist-test-slave-hhpq krb5kdc[29499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521912, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.252@KRBTEST.COM for kudu/127.7.17.252@KRBTEST.COM
I20260927 15:11:52.495563 29655 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } attempt: 3
Sep 27 15:11:52 dist-test-slave-hhpq krb5kdc[29499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.253@KRBTEST.COM
I20260927 15:11:52.498245 29580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12: Bootstrap starting.
I20260927 15:11:52.498950 29734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.490349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:34181 (local address 127.0.0.1:59746)
0927 15:11:52.490738 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.490750 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:52.490815 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:52.491224 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:52.491227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:52.491432 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:52.491514 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:52.491520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.492493 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.492496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:52.493316 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:52.493326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.493882 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.493885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:52.493963 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.494409 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:52.494422 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:52.495630 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:52.497544 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:52.497547 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:52.497549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:52.497770 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:52.498040 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:52.498043 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:52.498045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:52.498116 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:52.498505 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:52.498507 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:52.498584 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 15:11:52.498671 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 15:11:52.498791 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325}
I20260927 15:11:52.499397 29735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.490710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59746 (local address 127.7.17.252:34181)
0927 15:11:52.491053 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.491059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.491068 (+ 9us) server_negotiation.cc:407] Connection header received
0927 15:11:52.491093 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.491094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.491124 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.491198 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.491619 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.492387 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.493424 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.493730 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.493877 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.495762 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:52.495766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:52.495769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:52.495789 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:52.497429 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.497870 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.497873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.497875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.497954 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.498191 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.498194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.498196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.498392 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:52.498474 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:52.499125 (+ 651us) server_negotiation.cc:299] Negotiation successful
0927 15:11:52.499231 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":117,"threads_started":1}
I20260927 15:11:52.500392 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.490253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56694 (local address 127.7.17.253:41365)
0927 15:11:52.490361 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.490365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.490378 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:11:52.490470 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.490473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.490518 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.490598 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.491494 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.492271 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.493335 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.493594 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.493690 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.496706 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:52.496710 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:52.496712 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:52.496736 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:52.498218 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.498651 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.498655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.498657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.498733 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.499036 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.499039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.499040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.499237 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:52.499434 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:52.500140 (+ 706us) server_negotiation.cc:299] Negotiation successful
0927 15:11:52.500250 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 15:11:52 dist-test-slave-hhpq krb5kdc[29499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521912, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.253@KRBTEST.COM for kudu/127.7.17.252@KRBTEST.COM
I20260927 15:11:52.501567 29580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:52.502503 29730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84479b3c32174b28a672e4d9f8401ce0: Bootstrap starting.
I20260927 15:11:52.503036 29580 log.cc:824] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:52.504777 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:52.504928 29659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.495950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:34181 (local address 127.0.0.1:59748)
0927 15:11:52.496208 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:52.496222 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:52.496296 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:52.496572 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:52.496575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:52.496750 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:52.496874 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:52.496881 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.497811 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.497815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:52.498545 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:52.498556 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.499465 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.499470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:52.499557 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.499983 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:52.499997 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:52.501406 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:52.503327 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:52.503330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:52.503332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:52.503586 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:52.503902 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:52.503905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:52.503907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:52.503997 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:52.504422 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:52.504425 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:52.504522 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 15:11:52.504603 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 15:11:52.504739 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260927 15:11:52.505342 29730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84479b3c32174b28a672e4d9f8401ce0: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:52.505460 29580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12: No bootstrap required, opened a new log
I20260927 15:11:52.505864 29733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.496193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59748 (local address 127.7.17.252:34181)
0927 15:11:52.496367 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.496372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.496383 (+ 11us) server_negotiation.cc:407] Connection header received
0927 15:11:52.496415 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.496418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.496459 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.496535 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.496998 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.497693 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.498721 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.499317 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.500694 (+ 1377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.501545 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:52.501548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:52.501551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:52.501574 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:52.503205 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.503709 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.503712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.503714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.503796 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.504107 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.504109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.504111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.504308 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:52.505498 (+ 1190us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:52.505614 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 15:11:52.505726 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260927 15:11:52.506208 29730 log.cc:824] T 00000000000000000000000000000000 P 84479b3c32174b28a672e4d9f8401ce0: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:52.508172 29730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84479b3c32174b28a672e4d9f8401ce0: No bootstrap required, opened a new log
I20260927 15:11:52.508981 29554 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b3f939d9290c4b258bb8ebbfad06ef12"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42324
I20260927 15:11:52.509131 29655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db81012d9e3843dab6f157cdfe445e29: Bootstrap starting.
I20260927 15:11:52.509953 29580 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35127 } } peers { permanent_uuid: "db81012d9e3843dab6f157cdfe445e29" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } } peers { permanent_uuid: "84479b3c32174b28a672e4d9f8401ce0" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } }
I20260927 15:11:52.510164 29580 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f939d9290c4b258bb8ebbfad06ef12, State: Initialized, Role: FOLLOWER
I20260927 15:11:52.510502 29580 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35127 } } peers { permanent_uuid: "db81012d9e3843dab6f157cdfe445e29" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } } peers { permanent_uuid: "84479b3c32174b28a672e4d9f8401ce0" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } }
I20260927 15:11:52.511101 29738 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35127 } } peers { permanent_uuid: "db81012d9e3843dab6f157cdfe445e29" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } } peers { permanent_uuid: "84479b3c32174b28a672e4d9f8401ce0" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } } }
I20260927 15:11:52.511299 29738 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:11:52.511857 29655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db81012d9e3843dab6f157cdfe445e29: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:52.511920 29580 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:52.511967 29554 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 15:11:52.511831 29730 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84479b3c32174b28a672e4d9f8401ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35127 } } peers { permanent_uuid: "db81012d9e3843dab6f157cdfe445e29" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } } peers { permanent_uuid: "84479b3c32174b28a672e4d9f8401ce0" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } }
I20260927 15:11:52.512061 29730 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84479b3c32174b28a672e4d9f8401ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84479b3c32174b28a672e4d9f8401ce0, State: Initialized, Role: FOLLOWER
I20260927 15:11:52.512084 29554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:52.512383 29730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84479b3c32174b28a672e4d9f8401ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35127 } } peers { permanent_uuid: "db81012d9e3843dab6f157cdfe445e29" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } } peers { permanent_uuid: "84479b3c32174b28a672e4d9f8401ce0" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } }
I20260927 15:11:52.512718 29655 log.cc:824] T 00000000000000000000000000000000 P db81012d9e3843dab6f157cdfe445e29: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:52.513427 29554 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35127 } } peers { permanent_uuid: "db81012d9e3843dab6f157cdfe445e29" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } } peers { permanent_uuid: "84479b3c32174b28a672e4d9f8401ce0" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } }
I20260927 15:11:52.513739 29730 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84479b3c32174b28a672e4d9f8401ce0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:52.515488 29655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db81012d9e3843dab6f157cdfe445e29: No bootstrap required, opened a new log
I20260927 15:11:52.515515 29704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "84479b3c32174b28a672e4d9f8401ce0"
I20260927 15:11:52.515723 29704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84479b3c32174b28a672e4d9f8401ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:52.517503 29704 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 84479b3c32174b28a672e4d9f8401ce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3f939d9290c4b258bb8ebbfad06ef12 in term 1.
I20260927 15:11:52.519542 29655 raft_consensus.cc:348] T 00000000000000000000000000000000 P db81012d9e3843dab6f157cdfe445e29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35127 } } peers { permanent_uuid: "db81012d9e3843dab6f157cdfe445e29" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } } peers { permanent_uuid: "84479b3c32174b28a672e4d9f8401ce0" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } }
I20260927 15:11:52.519749 29655 raft_consensus.cc:729] T 00000000000000000000000000000000 P db81012d9e3843dab6f157cdfe445e29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db81012d9e3843dab6f157cdfe445e29, State: Initialized, Role: FOLLOWER
I20260927 15:11:52.520061 29655 consensus_queue.cc:260] T 00000000000000000000000000000000 P db81012d9e3843dab6f157cdfe445e29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35127 } } peers { permanent_uuid: "db81012d9e3843dab6f157cdfe445e29" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } } peers { permanent_uuid: "84479b3c32174b28a672e4d9f8401ce0" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } }
I20260927 15:11:52.514396 29554 leader_election.cc:288] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12 [CANDIDATE]: Term 1 election: Requested vote from peers db81012d9e3843dab6f157cdfe445e29 (127.7.17.253:41365), 84479b3c32174b28a672e4d9f8401ce0 (127.7.17.252:34181)
W20260927 15:11:52.521157 29763 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84479b3c32174b28a672e4d9f8401ce0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:11:52.521226 29763 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:11:52.517977 29531 leader_election.cc:302] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84479b3c32174b28a672e4d9f8401ce0, b3f939d9290c4b258bb8ebbfad06ef12; no voters:
I20260927 15:11:52.521531 29739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84479b3c32174b28a672e4d9f8401ce0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35127 } } peers { permanent_uuid: "db81012d9e3843dab6f157cdfe445e29" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } } peers { permanent_uuid: "84479b3c32174b28a672e4d9f8401ce0" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } } }
I20260927 15:11:52.521649 29739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84479b3c32174b28a672e4d9f8401ce0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:11:52.521843 29738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:52.522045 29738 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12 [term 1 LEADER]: Becoming Leader. State: Replica: b3f939d9290c4b258bb8ebbfad06ef12, State: Running, Role: LEADER
I20260927 15:11:52.522341 29738 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35127 } } peers { permanent_uuid: "db81012d9e3843dab6f157cdfe445e29" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } } peers { permanent_uuid: "84479b3c32174b28a672e4d9f8401ce0" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } }
I20260927 15:11:52.521214 29662 master_runner.cc:186] Error getting master registration for 127.7.17.253:41365: OK, instance_id { permanent_uuid: "db81012d9e3843dab6f157cdfe445e29" instance_seqno: 1790521912246043 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:11:52.524398 29662 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 15:11:52.519114 29751 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:11:52.524920 29751 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:11:52.525751 29765 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3f939d9290c4b258bb8ebbfad06ef12. Latest consensus state: current_term: 1 leader_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35127 } } peers { permanent_uuid: "db81012d9e3843dab6f157cdfe445e29" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } } peers { permanent_uuid: "84479b3c32174b28a672e4d9f8401ce0" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } } }
I20260927 15:11:52.525867 29765 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:52.527401 29771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:52.527412 29655 sys_catalog.cc:564] T 00000000000000000000000000000000 P db81012d9e3843dab6f157cdfe445e29 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:52.528447 29771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:52.530052 29772 sys_catalog.cc:455] T 00000000000000000000000000000000 P db81012d9e3843dab6f157cdfe445e29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35127 } } peers { permanent_uuid: "db81012d9e3843dab6f157cdfe445e29" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } } peers { permanent_uuid: "84479b3c32174b28a672e4d9f8401ce0" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } } }
I20260927 15:11:52.530122 29629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "db81012d9e3843dab6f157cdfe445e29"
I20260927 15:11:52.537710 29629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db81012d9e3843dab6f157cdfe445e29 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:52.527199 29512 master_runner.cc:186] Error getting master registration for 127.7.17.253:41365: OK, instance_id { permanent_uuid: "db81012d9e3843dab6f157cdfe445e29" instance_seqno: 1790521912246043 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:11:52.538566 29512 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:11:52.539702 29629 raft_consensus.cc:2454] T 00000000000000000000000000000000 P db81012d9e3843dab6f157cdfe445e29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3f939d9290c4b258bb8ebbfad06ef12 in term 1.
I20260927 15:11:52.540359 29772 sys_catalog.cc:458] T 00000000000000000000000000000000 P db81012d9e3843dab6f157cdfe445e29 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 15:11:52.545125 29784 catalog_manager.cc:1558] T 00000000000000000000000000000000 P db81012d9e3843dab6f157cdfe445e29: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:11:52.545222 29784 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:11:52.546372 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.526867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.7.17.253:41365)
0927 15:11:52.526980 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.526984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.530465 (+ 3481us) server_negotiation.cc:407] Connection header received
0927 15:11:52.530673 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.530676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.530726 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.530778 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.531815 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.532583 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.534086 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.534424 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.534504 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.540984 (+ 6480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:52.540989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:52.540993 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:52.541029 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:52.542824 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.543416 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.543420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.543421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.543514 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.545356 (+ 1842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.545360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.545362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.545595 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:52.545711 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:52.546077 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 15:11:52.546205 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 15:11:52.551214 29584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.540777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42326 (local address 127.7.17.254:35127)
0927 15:11:52.540932 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.540937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.542169 (+ 1232us) server_negotiation.cc:407] Connection header received
0927 15:11:52.542345 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.542349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.542400 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.542450 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.543429 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.544274 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.545630 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.545977 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.546051 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.547781 (+ 1730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:52.547785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:52.547788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:52.547816 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:52.549463 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.549932 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.549936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.549938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.550019 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.550285 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.550289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.550291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.550491 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:52.550597 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:52.550937 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 15:11:52.551055 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:11:52.559463 29733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:52.541548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59752 (local address 127.7.17.252:34181)
0927 15:11:52.541665 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.541670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.548057 (+ 6387us) server_negotiation.cc:407] Connection header received
0927 15:11:52.548226 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.548229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.548281 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.548329 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.549100 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.549857 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.553609 (+ 3752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:52.553978 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:52.554092 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:52.555846 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:52.555851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:52.555865 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:52.555897 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:52.557595 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.558095 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.558099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.558101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.558188 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:52.558474 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:52.558478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:52.558480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:52.558703 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:52.558812 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:52.559165 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 15:11:52.559288 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:11:52.565755 29704 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 84479b3c32174b28a672e4d9f8401ce0 [term 1 FOLLOWER]: Refusing update from remote peer b3f939d9290c4b258bb8ebbfad06ef12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:52.565972 29629 raft_consensus.cc:1261] T 00000000000000000000000000000000 P db81012d9e3843dab6f157cdfe445e29 [term 1 FOLLOWER]: Refusing update from remote peer b3f939d9290c4b258bb8ebbfad06ef12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:52.566598 29738 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84479b3c32174b28a672e4d9f8401ce0" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:52.566850 29765 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db81012d9e3843dab6f157cdfe445e29" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:52.571226 29739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84479b3c32174b28a672e4d9f8401ce0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3f939d9290c4b258bb8ebbfad06ef12. Latest consensus state: current_term: 1 leader_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35127 } } peers { permanent_uuid: "db81012d9e3843dab6f157cdfe445e29" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } } peers { permanent_uuid: "84479b3c32174b28a672e4d9f8401ce0" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } } }
I20260927 15:11:52.571426 29739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84479b3c32174b28a672e4d9f8401ce0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:11:52.571527 29772 sys_catalog.cc:455] T 00000000000000000000000000000000 P db81012d9e3843dab6f157cdfe445e29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3f939d9290c4b258bb8ebbfad06ef12. Latest consensus state: current_term: 1 leader_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35127 } } peers { permanent_uuid: "db81012d9e3843dab6f157cdfe445e29" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } } peers { permanent_uuid: "84479b3c32174b28a672e4d9f8401ce0" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } } }
I20260927 15:11:52.571652 29772 sys_catalog.cc:458] T 00000000000000000000000000000000 P db81012d9e3843dab6f157cdfe445e29 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:11:52.582685 29765 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35127 } } peers { permanent_uuid: "db81012d9e3843dab6f157cdfe445e29" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } } peers { permanent_uuid: "84479b3c32174b28a672e4d9f8401ce0" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } } }
I20260927 15:11:52.582847 29765 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:52.583004 29785 mvcc.cc:205] Tried to move back new op lower bound from 7333977753816788992 to 7333977753694539776. Current Snapshot: MvccSnapshot[applied={T|T < 7333977753816788992}]
I20260927 15:11:52.584403 29772 sys_catalog.cc:455] T 00000000000000000000000000000000 P db81012d9e3843dab6f157cdfe445e29 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35127 } } peers { permanent_uuid: "db81012d9e3843dab6f157cdfe445e29" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } } peers { permanent_uuid: "84479b3c32174b28a672e4d9f8401ce0" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } } }
I20260927 15:11:52.584532 29772 sys_catalog.cc:458] T 00000000000000000000000000000000 P db81012d9e3843dab6f157cdfe445e29 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:11:52.584638 29738 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35127 } } peers { permanent_uuid: "db81012d9e3843dab6f157cdfe445e29" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } } peers { permanent_uuid: "84479b3c32174b28a672e4d9f8401ce0" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } } }
I20260927 15:11:52.584740 29738 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:52.586009 29739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84479b3c32174b28a672e4d9f8401ce0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f939d9290c4b258bb8ebbfad06ef12" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 35127 } } peers { permanent_uuid: "db81012d9e3843dab6f157cdfe445e29" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 41365 } } peers { permanent_uuid: "84479b3c32174b28a672e4d9f8401ce0" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 34181 } } }
I20260927 15:11:52.586135 29739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84479b3c32174b28a672e4d9f8401ce0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:11:52.587435 29790 mvcc.cc:205] Tried to move back new op lower bound from 7333977753816788992 to 7333977753694539776. Current Snapshot: MvccSnapshot[applied={T|T < 7333977753816788992}]
I20260927 15:11:52.591118 29771 catalog_manager.cc:1347] Generated new cluster ID: 440e48611e6146eb853f013c3f4ecaa4
I20260927 15:11:52.591192 29771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:52.618170 29771 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:52.619513 29771 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:52.636427 29771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3f939d9290c4b258bb8ebbfad06ef12: Generated new TSK 0
I20260927 15:11:52.637427 29771 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:11:52.663956 29584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:52.660627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42338 (local address 127.7.17.254:35127)
0927 15:11:52.660762 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.660767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.662775 (+ 2008us) server_negotiation.cc:407] Connection header received
0927 15:11:52.663009 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.663013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.663067 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.663118 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.663797 (+ 679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42338: BlockingRecv error: recv got EOF from 127.0.0.1:42338 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 15:11:52.665351 29733 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:52.661436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59764 (local address 127.7.17.252:34181)
0927 15:11:52.661583 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.661589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.664494 (+ 2905us) server_negotiation.cc:407] Connection header received
0927 15:11:52.664613 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.664616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.664673 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.664772 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.665228 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59764: BlockingRecv error: recv got EOF from 127.0.0.1:59764 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 15:11:52.670123 29658 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:52.662091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.7.17.253:41365)
0927 15:11:52.662205 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.662209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.665069 (+ 2860us) server_negotiation.cc:407] Connection header received
0927 15:11:52.669247 (+ 4178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.669252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.669307 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.669360 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.669969 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56710: BlockingRecv error: recv got EOF from 127.0.0.1:56710 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 15:11:52.675014 29658 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:52.673360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.7.17.253:41365)
0927 15:11:52.673474 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.673479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.673988 (+ 509us) server_negotiation.cc:407] Connection header received
0927 15:11:52.674214 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.674218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.674294 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.674346 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.674864 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56714: BlockingRecv error: recv got EOF from 127.0.0.1:56714 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 15:11:52.680300 29658 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:52.678705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.7.17.253:41365)
0927 15:11:52.678811 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.678815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.679358 (+ 543us) server_negotiation.cc:407] Connection header received
0927 15:11:52.679565 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.679569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.679623 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.679672 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.680180 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56730: BlockingRecv error: recv got EOF from 127.0.0.1:56730 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 15:11:52.686295 29658 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:52.684489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.7.17.253:41365)
0927 15:11:52.684617 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.684622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.685293 (+ 671us) server_negotiation.cc:407] Connection header received
0927 15:11:52.685498 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.685501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.685555 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.685607 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.686167 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56736: BlockingRecv error: recv got EOF from 127.0.0.1:56736 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 15:11:52.695427 29658 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:52.693026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.7.17.253:41365)
0927 15:11:52.693189 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.693210 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.694109 (+ 899us) server_negotiation.cc:407] Connection header received
0927 15:11:52.694504 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.694508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.694569 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.694631 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.695293 (+ 662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56750: BlockingRecv error: recv got EOF from 127.0.0.1:56750 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260927 15:11:52.708350 29658 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:52.706154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56760 (local address 127.7.17.253:41365)
0927 15:11:52.706300 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.706305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.707072 (+ 767us) server_negotiation.cc:407] Connection header received
0927 15:11:52.707347 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.707351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.707433 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.707497 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.708222 (+ 725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56760: BlockingRecv error: recv got EOF from 127.0.0.1:56760 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 15:11:52.728986 29658 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:52.727304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56772 (local address 127.7.17.253:41365)
0927 15:11:52.727456 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.727462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.727928 (+ 466us) server_negotiation.cc:407] Connection header received
0927 15:11:52.728112 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.728115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.728178 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.728274 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.728805 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56772: BlockingRecv error: recv got EOF from 127.0.0.1:56772 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 15:11:52.764602 29658 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:52.763496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56782 (local address 127.7.17.253:41365)
0927 15:11:52.763644 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.763648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.763797 (+ 149us) server_negotiation.cc:407] Connection header received
0927 15:11:52.763933 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.763935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.763983 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.764054 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.764482 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56782: BlockingRecv error: recv got EOF from 127.0.0.1:56782 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 15:11:52.832032 29658 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:52.830877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56792 (local address 127.7.17.253:41365)
0927 15:11:52.831020 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.831025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.831178 (+ 153us) server_negotiation.cc:407] Connection header received
0927 15:11:52.831281 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.831284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.831330 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.831409 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.831906 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56792: BlockingRecv error: recv got EOF from 127.0.0.1:56792 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 15:11:52.963320 29658 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:52.962236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56796 (local address 127.7.17.253:41365)
0927 15:11:52.962391 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:52.962396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:52.962530 (+ 134us) server_negotiation.cc:407] Connection header received
0927 15:11:52.962656 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:52.962658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:52.962708 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:52.962789 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:52.963192 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56796: BlockingRecv error: recv got EOF from 127.0.0.1:56796 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 15:11:53.222584 29658 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:53.221428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56810 (local address 127.7.17.253:41365)
0927 15:11:53.221559 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:53.221563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:53.221717 (+ 154us) server_negotiation.cc:407] Connection header received
0927 15:11:53.221867 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:53.221870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:53.221923 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:53.221994 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:53.222463 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56810: BlockingRecv error: recv got EOF from 127.0.0.1:56810 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:11:53.522589 29763 catalog_manager.cc:1259] Loaded cluster ID: 440e48611e6146eb853f013c3f4ecaa4
I20260927 15:11:53.522684 29763 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 84479b3c32174b28a672e4d9f8401ce0: loading cluster ID for follower catalog manager: success
I20260927 15:11:53.524410 29763 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 84479b3c32174b28a672e4d9f8401ce0: acquiring CA information for follower catalog manager: success
I20260927 15:11:53.525699 29763 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 84479b3c32174b28a672e4d9f8401ce0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:11:53.546522 29784 catalog_manager.cc:1259] Loaded cluster ID: 440e48611e6146eb853f013c3f4ecaa4
I20260927 15:11:53.546607 29784 catalog_manager.cc:1552] T 00000000000000000000000000000000 P db81012d9e3843dab6f157cdfe445e29: loading cluster ID for follower catalog manager: success
I20260927 15:11:53.548310 29784 catalog_manager.cc:1574] T 00000000000000000000000000000000 P db81012d9e3843dab6f157cdfe445e29: acquiring CA information for follower catalog manager: success
I20260927 15:11:53.549508 29784 catalog_manager.cc:1602] T 00000000000000000000000000000000 P db81012d9e3843dab6f157cdfe445e29: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:11:53.740367 29861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:53.736562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56812 (local address 127.7.17.253:41365)
0927 15:11:53.736813 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:53.736817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:53.736866 (+ 49us) server_negotiation.cc:407] Connection header received
0927 15:11:53.737006 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:53.737009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:53.737139 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:53.737238 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:11:53.737470 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:53.738043 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:53.738915 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:53.739164 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:53.739250 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:53.739514 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:11:53.739971 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:11:53.740068 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:53.740139 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 15:11:53.740192 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":80,"threads_started":1}
I20260927 15:11:53.747187 29867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:53.743504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59774 (local address 127.7.17.252:34181)
0927 15:11:53.743744 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:53.743749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:53.743763 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:11:53.743894 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:53.743898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:53.744023 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:53.744140 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:11:53.744324 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:53.744979 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:53.745777 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:53.746003 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:53.746074 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:53.746340 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:11:53.746795 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:11:53.746887 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:53.746975 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 15:11:53.747027 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
I20260927 15:11:53.754652 29861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:53.750200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56818 (local address 127.7.17.253:41365)
0927 15:11:53.750348 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:53.750353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:53.750821 (+ 468us) server_negotiation.cc:407] Connection header received
0927 15:11:53.750966 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:53.750970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:53.751031 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:53.751098 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:11:53.751336 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:53.752006 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:53.752981 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:53.753515 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:53.753677 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:53.754152 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:11:53.754244 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:11:53.754290 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:53.754441 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 15:11:53.754499 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:11:53.755084 29867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:53.750646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59788 (local address 127.7.17.252:34181)
0927 15:11:53.750769 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:53.750773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:53.751043 (+ 270us) server_negotiation.cc:407] Connection header received
0927 15:11:53.751168 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:53.751171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:53.751217 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:53.751265 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:11:53.751705 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:53.752479 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:53.753918 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:53.754218 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:53.754279 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:53.754630 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:11:53.754702 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:11:53.754742 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:53.754900 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 15:11:53.754961 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:11:53.755332 29873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:53.750206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42348 (local address 127.7.17.254:35127)
0927 15:11:53.750461 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:53.750466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:53.750749 (+ 283us) server_negotiation.cc:407] Connection header received
0927 15:11:53.750923 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:53.750927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:53.751075 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:53.751416 (+ 341us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:11:53.751661 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:53.752488 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:53.753637 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:53.753988 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:53.754072 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:53.754390 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:11:53.754758 (+ 368us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:11:53.754837 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:53.755124 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 15:11:53.755192 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":81,"threads_started":1}
I20260927 15:11:53.764305 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 29512
I20260927 15:11:53.770078 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 29587
I20260927 15:11:53.775499 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 29662
2026-09-27T15:11:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3020 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:11:53 dist-test-slave-hhpq krb5kdc[29878](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:11:53 dist-test-slave-hhpq krb5kdc[29878](info): set up 2 sockets
Sep 27 15:11:53 dist-test-slave-hhpq krb5kdc[29878](info): commencing operation
krb5kdc: starting...
W20260927 15:11:54.729101 7239 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/test-user.keytab.
Sep 27 15:11:54 dist-test-slave-hhpq krb5kdc[29878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:11:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:11:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
Entry for principal kudu/127.7.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab.
I20260927 15:11:54.849453 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:36247
--webserver_interface=127.7.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:41933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:36247,127.7.17.253:38813,127.7.17.252:41451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:11:54.964473 29891 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:11:54.964813 29891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:54.964893 29891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:54.964958 29891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:11:54.969230 29891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:54.969321 29891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:54.969355 29891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:54.969376 29891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:54.969398 29891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:54.975021 29891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:41933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:36247,127.7.17.253:38813,127.7.17.252:41451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.254.keytab
--principal=kudu/127.7.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.254:36247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:54.976217 29891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:54.977349 29891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:54.986192 29896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:54.986209 29897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:54.986209 29899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:54.986356 29891 server_base.cc:1034] running on GCE node
I20260927 15:11:54.987143 29891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:54.988368 29891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:54.989522 29891 hybrid_clock.cc:648] HybridClock initialized: now 1790521914989504 us; error 33 us; skew 500 ppm
Sep 27 15:11:54 dist-test-slave-hhpq krb5kdc[29878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521914, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:54.992369 29891 init.cc:377] Logged in from keytab as kudu/127.7.17.254@KRBTEST.COM (short username kudu)
I20260927 15:11:54.993373 29891 webserver.cc:460] Webserver started at http://127.7.17.254:42083/ using document root <none> and password file <none>
I20260927 15:11:54.993927 29891 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:54.993994 29891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:54.994194 29891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:54.996491 29891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-0/data/instance:
uuid: "7a28524343f7453fa37399ca23704bb6"
format_stamp: "Formatted at 2026-09-27 15:11:54 on dist-test-slave-hhpq"
server_key: "ee4782ef0fba87e291c2c55bacbefcf4"
server_key_iv: "b2143171870fff297b870956acbcf444"
server_key_version: "encryptionkey@0"
I20260927 15:11:54.997105 29891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-0/wal/instance:
uuid: "7a28524343f7453fa37399ca23704bb6"
format_stamp: "Formatted at 2026-09-27 15:11:54 on dist-test-slave-hhpq"
server_key: "ee4782ef0fba87e291c2c55bacbefcf4"
server_key_iv: "b2143171870fff297b870956acbcf444"
server_key_version: "encryptionkey@0"
I20260927 15:11:55.001150 29891 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:11:55.003805 29906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:55.005321 29891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:55.005476 29891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-0/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-0/wal
uuid: "7a28524343f7453fa37399ca23704bb6"
format_stamp: "Formatted at 2026-09-27 15:11:54 on dist-test-slave-hhpq"
server_key: "ee4782ef0fba87e291c2c55bacbefcf4"
server_key_iv: "b2143171870fff297b870956acbcf444"
server_key_version: "encryptionkey@0"
I20260927 15:11:55.005596 29891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:55.027976 29891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:55.031422 29891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:55.031615 29891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:55.038794 29891 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.254:36247
I20260927 15:11:55.039152 29958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.254:36247 every 8 connection(s)
I20260927 15:11:55.039825 29891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-0/data/info.pb
I20260927 15:11:55.042725 29959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:55.046235 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 29891
I20260927 15:11:55.046341 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-0/wal/instance
I20260927 15:11:55.046591 7239 external_mini_cluster.cc:1428] Setting key c46da8c52590adc8bbe8ef718694d6de
I20260927 15:11:55.046794 29959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36247 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.17.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.253@KRBTEST.COM" created.
Sep 27 15:11:55 dist-test-slave-hhpq krb5kdc[29878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521914, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:11:55.065227 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.049305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:36247 (local address 127.0.0.1:54012)
0927 15:11:55.049820 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.049881 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:55.050949 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:55.051842 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:55.051847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:55.052216 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:55.052496 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:55.052510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.054378 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.054381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:55.055225 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:55.055236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.056158 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.056161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:55.056260 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.056988 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:55.057006 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:55.059639 (+ 2633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:55.061961 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:55.061974 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:55.061987 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:55.062327 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:55.062727 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:55.062731 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:55.062732 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:55.062900 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:55.063535 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:55.063543 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:55.063727 (+ 184us) client_negotiation.cc:769] Sending connection context
0927 15:11:55.063944 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 15:11:55.064358 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":73,"thread_start_us":125,"threads_started":1}
I20260927 15:11:55.065224 29963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.049305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.7.17.254:36247)
0927 15:11:55.049761 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.049780 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.049995 (+ 215us) server_negotiation.cc:407] Connection header received
0927 15:11:55.051377 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.051384 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.051603 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.051777 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.052659 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.054223 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.055363 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.055997 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.056136 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.059852 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:55.059860 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:55.059878 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:55.059908 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:55.061809 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.062461 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.062467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.062468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.062619 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.062992 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.062994 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.062995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.063416 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:55.063523 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:55.063989 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 15:11:55.064268 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":322,"spinlock_wait_cycles":34176,"thread_start_us":156,"threads_started":1}
I20260927 15:11:55.067947 29959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:11:55.068655 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.068399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:38813 (local address 127.0.0.1:50324)
0927 15:11:55.068517 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.068588 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.253:38813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 15:11:55.068924 29908 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.17.253:38813: connect: Connection refused (error 111)
I20260927 15:11:55.069757 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.069558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:38813 (local address 127.0.0.1:50328)
0927 15:11:55.069658 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.069704 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.253:38813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 15:11:55.069931 29959 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.253:38813: Network error: Client connection negotiation failed: client connection to 127.7.17.253:38813: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.253.keytab.
Entry for principal kudu/127.7.17.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.253.keytab.
I20260927 15:11:55.073786 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.253:38813
--webserver_interface=127.7.17.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.253.keytab
--principal=kudu/127.7.17.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:41933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:36247,127.7.17.253:38813,127.7.17.252:41451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:11:55.119212 29959 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } attempt: 1
I20260927 15:11:55.120127 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.119790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:38813 (local address 127.0.0.1:50334)
0927 15:11:55.119960 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.120036 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.253:38813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 15:11:55.120894 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.120647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:38813 (local address 127.0.0.1:50348)
0927 15:11:55.120743 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.120794 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.253:38813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 15:11:55.121022 29959 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.253:38813: Network error: Client connection negotiation failed: client connection to 127.7.17.253:38813: connect: Connection refused (error 111)
W20260927 15:11:55.185154 29966 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:11:55.185467 29966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:55.185488 29966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:55.185526 29966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 15:11:55.189175 29959 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } attempt: 2
I20260927 15:11:55.190017 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.189705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:38813 (local address 127.0.0.1:50362)
0927 15:11:55.189860 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.189929 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.253:38813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 15:11:55.190212 29966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:55.190277 29966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:55.190302 29966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:55.190332 29966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:55.190407 29966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:55.190727 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.190501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:38813 (local address 127.0.0.1:50364)
0927 15:11:55.190605 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.190657 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.253:38813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 15:11:55.190840 29959 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.253:38813: Network error: Client connection negotiation failed: client connection to 127.7.17.253:38813: connect: Connection refused (error 111)
I20260927 15:11:55.196215 29966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:41933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:36247,127.7.17.253:38813,127.7.17.252:41451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.253.keytab
--principal=kudu/127.7.17.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.253:38813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.17.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:55.197314 29966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:55.198292 29966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:11:55.206738 29971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:55.206738 29972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:55.206745 29974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:55.206961 29966 server_base.cc:1034] running on GCE node
I20260927 15:11:55.207641 29966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:55.208796 29966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:55.209993 29966 hybrid_clock.cc:648] HybridClock initialized: now 1790521915209964 us; error 38 us; skew 500 ppm
Sep 27 15:11:55 dist-test-slave-hhpq krb5kdc[29878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521915, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:55.212651 29966 init.cc:377] Logged in from keytab as kudu/127.7.17.253@KRBTEST.COM (short username kudu)
I20260927 15:11:55.213611 29966 webserver.cc:460] Webserver started at http://127.7.17.253:36677/ using document root <none> and password file <none>
I20260927 15:11:55.214159 29966 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:55.214221 29966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:55.214392 29966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:55.216573 29966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-1/data/instance:
uuid: "275c0d89025d4256a6c1922e3f6ddb4f"
format_stamp: "Formatted at 2026-09-27 15:11:55 on dist-test-slave-hhpq"
server_key: "973aa84757eb47bf60d7eaa228f9e0d9"
server_key_iv: "0ca097e11e6c272fc4cdd6e88de983d2"
server_key_version: "encryptionkey@0"
I20260927 15:11:55.217166 29966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-1/wal/instance:
uuid: "275c0d89025d4256a6c1922e3f6ddb4f"
format_stamp: "Formatted at 2026-09-27 15:11:55 on dist-test-slave-hhpq"
server_key: "973aa84757eb47bf60d7eaa228f9e0d9"
server_key_iv: "0ca097e11e6c272fc4cdd6e88de983d2"
server_key_version: "encryptionkey@0"
I20260927 15:11:55.221060 29966 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:11:55.223443 29981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:55.224695 29966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:11:55.224907 29966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-1/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-1/wal
uuid: "275c0d89025d4256a6c1922e3f6ddb4f"
format_stamp: "Formatted at 2026-09-27 15:11:55 on dist-test-slave-hhpq"
server_key: "973aa84757eb47bf60d7eaa228f9e0d9"
server_key_iv: "0ca097e11e6c272fc4cdd6e88de983d2"
server_key_version: "encryptionkey@0"
I20260927 15:11:55.225023 29966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:55.241670 29966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:55.245446 29966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:55.245653 29966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:55.252600 29966 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.253:38813
I20260927 15:11:55.252872 30033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.253:38813 every 8 connection(s)
I20260927 15:11:55.253650 29966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-1/data/info.pb
I20260927 15:11:55.256815 30034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:55.260130 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 29966
I20260927 15:11:55.260255 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-1/wal/instance
I20260927 15:11:55.260519 7239 external_mini_cluster.cc:1428] Setting key bd10826d7dc16d954afdc08802d3caf3
I20260927 15:11:55.261323 30034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36247 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.17.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.17.252@KRBTEST.COM" created.
Sep 27 15:11:55 dist-test-slave-hhpq krb5kdc[29878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521915, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.253@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:11:55.275439 29963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.263415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54028 (local address 127.7.17.254:36247)
0927 15:11:55.263568 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.263573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.264514 (+ 941us) server_negotiation.cc:407] Connection header received
0927 15:11:55.265560 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.265564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.265629 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.265731 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.266965 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.267792 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.269037 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.269291 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.269377 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.271906 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:55.271909 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:55.271912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:55.271940 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:55.273361 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.273883 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.273886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.273888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.273964 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.274293 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.274297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.274298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.274497 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:55.274583 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:55.275132 (+ 549us) server_negotiation.cc:299] Negotiation successful
0927 15:11:55.275266 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:11:55.276340 30037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.263829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:36247 (local address 127.0.0.1:54028)
0927 15:11:55.264332 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.264388 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:55.265341 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:55.265938 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:55.265945 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:55.266297 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:55.266795 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:55.266809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.267944 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.267947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:55.268919 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:55.268930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.269434 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.269436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:55.269586 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.270177 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:55.270194 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:55.271701 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:55.273483 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:55.273490 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:55.273508 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:55.273779 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:55.274048 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:55.274051 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:55.274054 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:55.274204 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:55.274595 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:55.274603 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:55.274884 (+ 281us) client_negotiation.cc:769] Sending connection context
0927 15:11:55.275064 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 15:11:55.275441 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":136,"threads_started":1}
I20260927 15:11:55.277884 30034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:11:55.281978 29959 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } attempt: 3
Sep 27 15:11:55 dist-test-slave-hhpq krb5kdc[29878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521915, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.253@KRBTEST.COM for kudu/127.7.17.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.17.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.252.keytab.
Entry for principal kudu/127.7.17.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.252.keytab.
Sep 27 15:11:55 dist-test-slave-hhpq krb5kdc[29878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521914, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for kudu/127.7.17.253@KRBTEST.COM
I20260927 15:11:55.287518 30037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.278384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:38813 (local address 127.0.0.1:50372)
0927 15:11:55.278520 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.278537 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:55.278618 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:55.279335 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:55.279338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:55.279474 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:55.279566 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:55.279571 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.280633 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.280636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:55.281230 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:55.281239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.281785 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.281787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:55.281881 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.282369 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:55.282380 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:55.283732 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:55.285874 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:55.285878 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:55.285880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:55.286097 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:55.286439 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:55.286442 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:55.286444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:55.286529 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:55.286983 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:55.286987 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:55.287083 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 15:11:55.287185 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 15:11:55.287316 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 15:11:55.287921 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.278572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50372 (local address 127.7.17.253:38813)
0927 15:11:55.278972 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.278981 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.279001 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:11:55.279036 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.279041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.279204 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.279313 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.279651 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.280542 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.281324 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.281696 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.281778 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.283846 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:55.283857 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:55.283873 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:55.283898 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:55.285740 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.286202 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.286209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.286211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.286295 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.286608 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.286610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.286611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.286880 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:55.287016 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:55.287294 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 15:11:55.287486 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":192,"threads_started":1}
I20260927 15:11:55.288050 7239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
/tmp/dist-test-taskgkjf1n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.252:41451
--webserver_interface=127.7.17.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.252.keytab
--principal=kudu/127.7.17.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.17.212:41933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.17.254:36247,127.7.17.253:38813,127.7.17.252:41451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:11:55.290431 30034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:11:55.291172 30037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.290901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:41451 (local address 127.0.0.1:58606)
0927 15:11:55.291021 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.291098 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:41451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 15:11:55.291358 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.282427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:38813 (local address 127.0.0.1:50382)
0927 15:11:55.282545 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.282560 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:55.282640 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:55.283228 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:55.283232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:55.283411 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:55.283525 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:55.283533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.284499 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.284502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:55.285113 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:55.285124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.285739 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.285742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:55.285843 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.286224 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:55.286236 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:55.287582 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:55.289890 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:55.289894 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:55.289896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:55.290135 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:55.290387 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:55.290390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:55.290392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:55.290471 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:55.290809 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:55.290813 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:55.290928 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 15:11:55.291039 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 15:11:55.291195 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 15:11:55.291373 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.282617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50382 (local address 127.7.17.253:38813)
0927 15:11:55.282991 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.283001 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.283014 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:11:55.283047 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.283049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.283089 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.283162 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.283624 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.284390 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.285240 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.285570 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.285710 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.288219 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:55.288225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:55.288229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:55.288253 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:55.289755 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.290232 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.290235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.290236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.290299 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.290551 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.290554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.290556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.290725 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:55.290811 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:55.291103 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 15:11:55.291203 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":97,"threads_started":1}
W20260927 15:11:55.291440 29985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.17.252:41451: connect: Connection refused (error 111)
I20260927 15:11:55.292037 29959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:11:55.292563 30037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.292298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:41451 (local address 127.0.0.1:58608)
0927 15:11:55.292452 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.292498 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:41451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 15:11:55.292672 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.292445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:41451 (local address 127.0.0.1:58620)
0927 15:11:55.292549 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.292598 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:41451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 15:11:55.292733 30034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.252:41451: Network error: Client connection negotiation failed: client connection to 127.7.17.252:41451: connect: Connection refused (error 111)
I20260927 15:11:55.293612 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.293356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:41451 (local address 127.0.0.1:58622)
0927 15:11:55.293496 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.293542 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:41451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260927 15:11:55.293725 29959 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.252:41451: Network error: Client connection negotiation failed: client connection to 127.7.17.252:41451: connect: Connection refused (error 111)
I20260927 15:11:55.324905 29959 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } attempt: 1
I20260927 15:11:55.325688 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.325405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:41451 (local address 127.0.0.1:58636)
0927 15:11:55.325547 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.325611 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:41451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 15:11:55.326452 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.326201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:41451 (local address 127.0.0.1:58638)
0927 15:11:55.326307 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.326358 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:41451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 15:11:55.326588 29959 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.252:41451: Network error: Client connection negotiation failed: client connection to 127.7.17.252:41451: connect: Connection refused (error 111)
I20260927 15:11:55.342053 30034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } attempt: 1
I20260927 15:11:55.342945 30037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.342627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:41451 (local address 127.0.0.1:58654)
0927 15:11:55.342796 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.342867 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:41451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 15:11:55.343703 30037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.343479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:41451 (local address 127.0.0.1:58670)
0927 15:11:55.343591 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.343640 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:41451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 15:11:55.343875 30034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.252:41451: Network error: Client connection negotiation failed: client connection to 127.7.17.252:41451: connect: Connection refused (error 111)
W20260927 15:11:55.400528 30041 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:11:55.400870 30041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:11:55.400913 30041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:11:55.400976 30041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 15:11:55.401739 29959 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } attempt: 2
I20260927 15:11:55.402508 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.402223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:41451 (local address 127.0.0.1:58680)
0927 15:11:55.402366 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.402428 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:41451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 15:11:55.403264 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.403034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:41451 (local address 127.0.0.1:58686)
0927 15:11:55.403142 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.403196 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:41451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 15:11:55.403412 29959 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.252:41451: Network error: Client connection negotiation failed: client connection to 127.7.17.252:41451: connect: Connection refused (error 111)
W20260927 15:11:55.405313 30041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:11:55.405362 30041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:11:55.405381 30041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:11:55.405404 30041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:11:55.405423 30041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:11:55.411129 30041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.17.212:41933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.17.254:36247,127.7.17.253:38813,127.7.17.252:41451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/krb5kdc/kudu_127.7.17.252.keytab
--principal=kudu/127.7.17.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.17.252:41451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.17.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:11:55.412031 30034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } attempt: 2
I20260927 15:11:55.412199 30041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:11:55.412801 30037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.412525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:41451 (local address 127.0.0.1:58692)
0927 15:11:55.412663 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.412725 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:41451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 15:11:55.413421 30041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:11:55.413538 30037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.413305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:41451 (local address 127.0.0.1:58696)
0927 15:11:55.413414 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.413464 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.17.252:41451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 15:11:55.413672 30034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.17.252:41451: Network error: Client connection negotiation failed: client connection to 127.7.17.252:41451: connect: Connection refused (error 111)
W20260927 15:11:55.422333 30050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:11:55.422362 30048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:55.422536 30041 server_base.cc:1034] running on GCE node
W20260927 15:11:55.422333 30047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:11:55.423336 30041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:11:55.424451 30041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:11:55.425625 30041 hybrid_clock.cc:648] HybridClock initialized: now 1790521915425605 us; error 33 us; skew 500 ppm
Sep 27 15:11:55 dist-test-slave-hhpq krb5kdc[29878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521915, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:11:55.428434 30041 init.cc:377] Logged in from keytab as kudu/127.7.17.252@KRBTEST.COM (short username kudu)
I20260927 15:11:55.429426 30041 webserver.cc:460] Webserver started at http://127.7.17.252:35363/ using document root <none> and password file <none>
I20260927 15:11:55.429979 30041 fs_manager.cc:359] Metadata directory not provided
I20260927 15:11:55.430045 30041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:11:55.430244 30041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:11:55.432294 30041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-2/data/instance:
uuid: "d361afcc93fd4edcbf5c44821a8eba98"
format_stamp: "Formatted at 2026-09-27 15:11:55 on dist-test-slave-hhpq"
server_key: "9817d8bad09ef7563bf34ea87700844e"
server_key_iv: "93de97b9649ad96c2bd9104aef4e3af3"
server_key_version: "encryptionkey@0"
I20260927 15:11:55.432858 30041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-2/wal/instance:
uuid: "d361afcc93fd4edcbf5c44821a8eba98"
format_stamp: "Formatted at 2026-09-27 15:11:55 on dist-test-slave-hhpq"
server_key: "9817d8bad09ef7563bf34ea87700844e"
server_key_iv: "93de97b9649ad96c2bd9104aef4e3af3"
server_key_version: "encryptionkey@0"
I20260927 15:11:55.436580 30041 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:11:55.439215 30057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:11:55.440586 30041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:11:55.440719 30041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-2/data,/tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-2/wal
uuid: "d361afcc93fd4edcbf5c44821a8eba98"
format_stamp: "Formatted at 2026-09-27 15:11:55 on dist-test-slave-hhpq"
server_key: "9817d8bad09ef7563bf34ea87700844e"
server_key_iv: "93de97b9649ad96c2bd9104aef4e3af3"
server_key_version: "encryptionkey@0"
I20260927 15:11:55.440877 30041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:11:55.471781 30041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:11:55.475144 30041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:11:55.475328 30041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:11:55.482280 30041 rpc_server.cc:307] RPC server started. Bound to: 127.7.17.252:41451
I20260927 15:11:55.482546 30109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.17.252:41451 every 8 connection(s)
I20260927 15:11:55.483309 30041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-2/data/info.pb
I20260927 15:11:55.485603 7239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu as pid 30041
I20260927 15:11:55.485738 7239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgkjf1n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521636012660-7239-0/minicluster-data/master-2/wal/instance
I20260927 15:11:55.486058 7239 external_mini_cluster.cc:1428] Setting key b23df290fab4dd7c11d964825d2aae64
I20260927 15:11:55.486531 30110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:11:55.491246 30110 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36247 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:11:55 dist-test-slave-hhpq krb5kdc[29878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.252@KRBTEST.COM
Sep 27 15:11:55 dist-test-slave-hhpq krb5kdc[29878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521915, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.252@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
I20260927 15:11:55.501216 30113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.487449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.7.17.252:41451)
0927 15:11:55.487860 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.487876 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.487916 (+ 40us) server_negotiation.cc:407] Connection header received
0927 15:11:55.488604 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.488623 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.489039 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.489355 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.490275 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.491278 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.492419 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.492957 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.493088 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.495662 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:55.495671 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:55.495686 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:55.495718 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:55.497721 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.498212 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.498219 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.498226 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.498366 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.498682 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.498685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.498686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.499177 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:55.499323 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:55.499701 (+ 378us) server_negotiation.cc:299] Negotiation successful
0927 15:11:55.500013 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":120,"threads_started":1}
I20260927 15:11:55.502547 29959 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } attempt: 3
I20260927 15:11:55.503732 29963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.493272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54040 (local address 127.7.17.254:36247)
0927 15:11:55.493400 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.493404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.494008 (+ 604us) server_negotiation.cc:407] Connection header received
0927 15:11:55.494131 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.494134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.494186 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.494252 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.494966 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.495571 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.496910 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.497254 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.497377 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.499575 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:55.499580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:55.499583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:55.499613 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:55.501765 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.502278 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.502282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.502283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.502389 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.502682 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.502685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.502687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.502921 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:55.503014 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:55.503439 (+ 425us) server_negotiation.cc:299] Negotiation successful
0927 15:11:55.503571 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:11:55.504724 30114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.493219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.254:36247 (local address 127.0.0.1:54040)
0927 15:11:55.493878 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.493921 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:55.494040 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:55.494317 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:55.494323 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:55.494689 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:55.494827 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:55.494842 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.495733 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.495736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:55.496733 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:55.496746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.497405 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.497409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:55.497505 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.497985 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:55.497996 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:55.499414 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:55.501929 (+ 2515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:55.501937 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:55.501942 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:55.502179 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:55.502486 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:55.502488 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:55.502490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:55.502585 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:55.503066 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:55.503072 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:55.503198 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 15:11:55.504345 (+ 1147us) client_negotiation.cc:240] Negotiation successful
0927 15:11:55.504534 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":175,"threads_started":1}
I20260927 15:11:55.504863 30034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } attempt: 3
I20260927 15:11:55.506089 30110 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:11:55 dist-test-slave-hhpq krb5kdc[29878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521914, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.254@KRBTEST.COM for kudu/127.7.17.252@KRBTEST.COM
Sep 27 15:11:55 dist-test-slave-hhpq krb5kdc[29878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.254@KRBTEST.COM
Sep 27 15:11:55 dist-test-slave-hhpq krb5kdc[29878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521915, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.253@KRBTEST.COM for kudu/127.7.17.252@KRBTEST.COM
I20260927 15:11:55.512328 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.502979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:41451 (local address 127.0.0.1:58708)
0927 15:11:55.503188 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.503203 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:55.503274 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:55.503791 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:55.503794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:55.503954 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:55.504054 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:55.504061 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.505124 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.505127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:55.505745 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:55.505754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.506344 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.506348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:55.506424 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.507004 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:55.507017 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:55.508514 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:55.510658 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:55.510662 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:55.510664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:55.510919 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:55.511248 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:55.511251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:55.511253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:55.511345 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:55.511809 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:55.511812 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:55.511915 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 15:11:55.511999 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 15:11:55.512140 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
Sep 27 15:11:55 dist-test-slave-hhpq krb5kdc[29878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521915, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.252@KRBTEST.COM for kudu/127.7.17.253@KRBTEST.COM
I20260927 15:11:55.513329 30113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.503043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58708 (local address 127.7.17.252:41451)
0927 15:11:55.503577 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.503581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.503593 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:11:55.503628 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.503631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.503672 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.504162 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.504245 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.505012 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.505878 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.506227 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.508652 (+ 2425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.508766 (+ 114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:55.508769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:55.508772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:55.508789 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:55.510530 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.511046 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.511049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.511051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.511130 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.511457 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.511460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.511462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.511693 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:55.512907 (+ 1214us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:55.512995 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 15:11:55.513171 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445}
I20260927 15:11:55.516149 29963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.503798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54054 (local address 127.7.17.254:36247)
0927 15:11:55.504131 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.504135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.504148 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:11:55.504176 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.504179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.504212 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.504284 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.505708 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.506379 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.507521 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.507845 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.508919 (+ 1074us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.512444 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:55.512448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:55.512450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:55.512474 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:55.514260 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.514870 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.514873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.514875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.514954 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.515240 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.515244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.515245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.515478 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:55.515712 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:55.515856 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 15:11:55.515976 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":71,"threads_started":1}
I20260927 15:11:55.516392 30037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.505255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:41451 (local address 127.0.0.1:58722)
0927 15:11:55.505475 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.505488 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:55.505576 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:55.506299 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:55.506302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:55.506479 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:55.506585 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:55.506593 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.507774 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.507778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:55.508618 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:55.508632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.510258 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.510262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:55.510354 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.510768 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:55.510783 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:55.512111 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:55.514856 (+ 2745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:55.514860 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:55.514866 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:55.515076 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:55.515366 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:55.515370 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:55.515371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:55.515459 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:55.515902 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:55.515905 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:55.516016 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 15:11:55.516097 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 15:11:55.516234 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":2}
I20260927 15:11:55.517309 29959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6: Bootstrap starting.
I20260927 15:11:55.517493 30116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.505777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58722 (local address 127.7.17.252:41451)
0927 15:11:55.506065 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.506072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.506085 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:11:55.506116 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.506119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.506158 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.506707 (+ 549us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.506794 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.507662 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.509696 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.510130 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.510555 (+ 425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.512290 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:55.512295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:55.512298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:55.512325 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:55.514732 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.515177 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.515180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.515182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.515270 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.515551 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.515554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.515556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.515781 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:55.517083 (+ 1302us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:55.517174 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 15:11:55.517305 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":216,"thread_start_us":96,"threads_started":1}
I20260927 15:11:55.519708 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.507017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50386 (local address 127.7.17.253:38813)
0927 15:11:55.507586 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.507590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.507601 (+ 11us) server_negotiation.cc:407] Connection header received
0927 15:11:55.507635 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.507638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.507676 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.507739 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.508756 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.509542 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.510694 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.511039 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.511132 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.513374 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:55.513378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:55.513381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:55.513406 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:55.517491 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.518264 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.518267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.518269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.518398 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.518741 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.518744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.518746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.519010 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:55.519077 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:55.519434 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 15:11:55.519562 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":903,"server-negotiator.queue_time_us":491}
I20260927 15:11:55.520016 29959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:55.520414 30114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.506774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.253:38813 (local address 127.0.0.1:50386)
0927 15:11:55.506939 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.506951 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:55.507014 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:55.507838 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:55.507840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:55.508014 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:55.508529 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:55.508538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.509649 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.509653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:55.510553 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:55.510613 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.511242 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.511246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:55.511339 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.511728 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:55.511742 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:55.513205 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:55.517885 (+ 4680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:55.517888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:55.517890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:55.518147 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:55.518536 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:55.518539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:55.518541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:55.518639 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:55.519158 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:55.519161 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:55.519256 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 15:11:55.520097 (+ 841us) client_negotiation.cc:240] Negotiation successful
0927 15:11:55.520249 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":1}
I20260927 15:11:55.520931 29959 log.cc:824] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:55.520977 30034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 275c0d89025d4256a6c1922e3f6ddb4f: Bootstrap starting.
I20260927 15:11:55.521150 30110 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:11:55.523443 29959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6: No bootstrap required, opened a new log
I20260927 15:11:55.524082 30034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 275c0d89025d4256a6c1922e3f6ddb4f: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:55.525009 30034 log.cc:824] T 00000000000000000000000000000000 P 275c0d89025d4256a6c1922e3f6ddb4f: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:11:55 dist-test-slave-hhpq krb5kdc[29878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521915, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.17.252@KRBTEST.COM for kudu/127.7.17.252@KRBTEST.COM
Sep 27 15:11:55 dist-test-slave-hhpq krb5kdc[29878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.17.253@KRBTEST.COM
I20260927 15:11:55.527734 29959 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a28524343f7453fa37399ca23704bb6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36247 } } peers { permanent_uuid: "275c0d89025d4256a6c1922e3f6ddb4f" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } } peers { permanent_uuid: "d361afcc93fd4edcbf5c44821a8eba98" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } }
I20260927 15:11:55.527995 29959 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a28524343f7453fa37399ca23704bb6, State: Initialized, Role: FOLLOWER
I20260927 15:11:55.528363 29959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a28524343f7453fa37399ca23704bb6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36247 } } peers { permanent_uuid: "275c0d89025d4256a6c1922e3f6ddb4f" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } } peers { permanent_uuid: "d361afcc93fd4edcbf5c44821a8eba98" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } }
I20260927 15:11:55.529117 30034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 275c0d89025d4256a6c1922e3f6ddb4f: No bootstrap required, opened a new log
I20260927 15:11:55.530308 29959 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:55.530519 30114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.521638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.17.252:41451 (local address 127.0.0.1:58736)
0927 15:11:55.521723 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 15:11:55.521735 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:11:55.521803 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:11:55.522546 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:11:55.522548 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:11:55.522718 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:11:55.522802 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:55.522808 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.523718 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.523721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:55.524439 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:11:55.524449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.525118 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.525121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:11:55.525193 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.525578 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:11:55.525592 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:11:55.527123 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:11:55.529061 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:55.529064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:55.529066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:55.529298 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:55.529565 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:11:55.529567 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:11:55.529569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:11:55.529637 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:11:55.530052 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:11:55.530055 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:11:55.530133 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 15:11:55.530223 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 15:11:55.530358 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 15:11:55.531301 30116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.521755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58736 (local address 127.7.17.252:41451)
0927 15:11:55.522312 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.522317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.522328 (+ 11us) server_negotiation.cc:407] Connection header received
0927 15:11:55.522359 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.522361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.522400 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.522462 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.522908 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.523593 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.524550 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.524936 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.525024 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.527266 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:55.527270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:55.527273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:55.527295 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:55.528961 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.529412 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.529415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.529417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.529486 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.529728 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.529730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.529732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.529941 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:55.530023 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:55.531028 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0927 15:11:55.531145 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485}
I20260927 15:11:55.531935 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.518385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50394 (local address 127.7.17.253:38813)
0927 15:11:55.518502 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.518512 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.521081 (+ 2569us) server_negotiation.cc:407] Connection header received
0927 15:11:55.521279 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.521282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.521330 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.521378 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.522258 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.523020 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.524094 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.524428 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.524502 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.527874 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:55.527878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:55.527881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:55.527907 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:55.529483 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.530083 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.530087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.530089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.530185 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.530518 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.530521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.530522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.530741 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:55.530959 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:55.531653 (+ 694us) server_negotiation.cc:299] Negotiation successful
0927 15:11:55.531778 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 15:11:55.533236 29912 tablet.cc:2369] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:11:55.534008 30119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a28524343f7453fa37399ca23704bb6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36247 } } peers { permanent_uuid: "275c0d89025d4256a6c1922e3f6ddb4f" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } } peers { permanent_uuid: "d361afcc93fd4edcbf5c44821a8eba98" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } } }
I20260927 15:11:55.534242 30119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:11:55.534795 30110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d361afcc93fd4edcbf5c44821a8eba98: Bootstrap starting.
W20260927 15:11:55.536432 30130 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:11:55.536732 30130 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:11:55.537621 30110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d361afcc93fd4edcbf5c44821a8eba98: Neither blocks nor log segments found. Creating new log.
I20260927 15:11:55.538504 30110 log.cc:824] T 00000000000000000000000000000000 P d361afcc93fd4edcbf5c44821a8eba98: Log is configured to *not* fsync() on all Append() calls
I20260927 15:11:55.536346 29891 master_runner.cc:186] Error getting master registration for 127.7.17.253:38813: OK, instance_id { permanent_uuid: "275c0d89025d4256a6c1922e3f6ddb4f" instance_seqno: 1790521915250636 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:11:55.539903 29891 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:11:55.540652 30110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d361afcc93fd4edcbf5c44821a8eba98: No bootstrap required, opened a new log
I20260927 15:11:55.540807 30034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 275c0d89025d4256a6c1922e3f6ddb4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a28524343f7453fa37399ca23704bb6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36247 } } peers { permanent_uuid: "275c0d89025d4256a6c1922e3f6ddb4f" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } } peers { permanent_uuid: "d361afcc93fd4edcbf5c44821a8eba98" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } }
I20260927 15:11:55.541105 30034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 275c0d89025d4256a6c1922e3f6ddb4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 275c0d89025d4256a6c1922e3f6ddb4f, State: Initialized, Role: FOLLOWER
I20260927 15:11:55.541533 30034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 275c0d89025d4256a6c1922e3f6ddb4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a28524343f7453fa37399ca23704bb6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36247 } } peers { permanent_uuid: "275c0d89025d4256a6c1922e3f6ddb4f" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } } peers { permanent_uuid: "d361afcc93fd4edcbf5c44821a8eba98" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } }
I20260927 15:11:55.542248 7239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:11:55.542376 30133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 275c0d89025d4256a6c1922e3f6ddb4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a28524343f7453fa37399ca23704bb6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36247 } } peers { permanent_uuid: "275c0d89025d4256a6c1922e3f6ddb4f" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } } peers { permanent_uuid: "d361afcc93fd4edcbf5c44821a8eba98" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } } }
I20260927 15:11:55.542526 30133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 275c0d89025d4256a6c1922e3f6ddb4f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:11:55.543085 30034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 275c0d89025d4256a6c1922e3f6ddb4f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:55.543560 29933 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7a28524343f7453fa37399ca23704bb6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54054
I20260927 15:11:55.543835 29933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 15:11:55.543941 29933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:55.544524 30110 raft_consensus.cc:348] T 00000000000000000000000000000000 P d361afcc93fd4edcbf5c44821a8eba98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a28524343f7453fa37399ca23704bb6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36247 } } peers { permanent_uuid: "275c0d89025d4256a6c1922e3f6ddb4f" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } } peers { permanent_uuid: "d361afcc93fd4edcbf5c44821a8eba98" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } }
I20260927 15:11:55.544777 30110 raft_consensus.cc:729] T 00000000000000000000000000000000 P d361afcc93fd4edcbf5c44821a8eba98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d361afcc93fd4edcbf5c44821a8eba98, State: Initialized, Role: FOLLOWER
I20260927 15:11:55.545293 30110 consensus_queue.cc:260] T 00000000000000000000000000000000 P d361afcc93fd4edcbf5c44821a8eba98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a28524343f7453fa37399ca23704bb6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36247 } } peers { permanent_uuid: "275c0d89025d4256a6c1922e3f6ddb4f" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } } peers { permanent_uuid: "d361afcc93fd4edcbf5c44821a8eba98" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } }
I20260927 15:11:55.545369 29933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a28524343f7453fa37399ca23704bb6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36247 } } peers { permanent_uuid: "275c0d89025d4256a6c1922e3f6ddb4f" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } } peers { permanent_uuid: "d361afcc93fd4edcbf5c44821a8eba98" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } }
I20260927 15:11:55.546917 30110 sys_catalog.cc:564] T 00000000000000000000000000000000 P d361afcc93fd4edcbf5c44821a8eba98 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:11:55.547326 30084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a28524343f7453fa37399ca23704bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d361afcc93fd4edcbf5c44821a8eba98"
I20260927 15:11:55.547541 30084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d361afcc93fd4edcbf5c44821a8eba98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:55.549636 30008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a28524343f7453fa37399ca23704bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "275c0d89025d4256a6c1922e3f6ddb4f"
I20260927 15:11:55.549883 30008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 275c0d89025d4256a6c1922e3f6ddb4f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:11:55.551215 30008 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 275c0d89025d4256a6c1922e3f6ddb4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a28524343f7453fa37399ca23704bb6 in term 1.
I20260927 15:11:55.551734 29908 leader_election.cc:302] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 275c0d89025d4256a6c1922e3f6ddb4f, 7a28524343f7453fa37399ca23704bb6; no voters:
I20260927 15:11:55.552039 30119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:11:55.552259 30119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6 [term 1 LEADER]: Becoming Leader. State: Replica: 7a28524343f7453fa37399ca23704bb6, State: Running, Role: LEADER
W20260927 15:11:55.552261 30144 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 275c0d89025d4256a6c1922e3f6ddb4f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:11:55.552323 30144 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:11:55.552338 30084 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d361afcc93fd4edcbf5c44821a8eba98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a28524343f7453fa37399ca23704bb6 in term 1.
I20260927 15:11:55.552610 30119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a28524343f7453fa37399ca23704bb6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36247 } } peers { permanent_uuid: "275c0d89025d4256a6c1922e3f6ddb4f" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } } peers { permanent_uuid: "d361afcc93fd4edcbf5c44821a8eba98" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } }
I20260927 15:11:55.552734 29933 leader_election.cc:288] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6 [CANDIDATE]: Term 1 election: Requested vote from peers 275c0d89025d4256a6c1922e3f6ddb4f (127.7.17.253:38813), d361afcc93fd4edcbf5c44821a8eba98 (127.7.17.252:41451)
I20260927 15:11:55.553155 29966 master_runner.cc:186] Error getting master registration for 127.7.17.252:41451: OK, instance_id { permanent_uuid: "d361afcc93fd4edcbf5c44821a8eba98" instance_seqno: 1790521915480286 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:11:55.556458 29966 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:11:55.557201 30119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a28524343f7453fa37399ca23704bb6. Latest consensus state: current_term: 1 leader_uuid: "7a28524343f7453fa37399ca23704bb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a28524343f7453fa37399ca23704bb6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36247 } } peers { permanent_uuid: "275c0d89025d4256a6c1922e3f6ddb4f" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } } peers { permanent_uuid: "d361afcc93fd4edcbf5c44821a8eba98" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } } }
I20260927 15:11:55.557343 30119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:55.560127 30145 sys_catalog.cc:455] T 00000000000000000000000000000000 P d361afcc93fd4edcbf5c44821a8eba98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a28524343f7453fa37399ca23704bb6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36247 } } peers { permanent_uuid: "275c0d89025d4256a6c1922e3f6ddb4f" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } } peers { permanent_uuid: "d361afcc93fd4edcbf5c44821a8eba98" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } } }
I20260927 15:11:55.563967 30145 sys_catalog.cc:458] T 00000000000000000000000000000000 P d361afcc93fd4edcbf5c44821a8eba98 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 15:11:55.563868 30158 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d361afcc93fd4edcbf5c44821a8eba98: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:11:55.564162 30158 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:11:55.565737 30116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.555241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58746 (local address 127.7.17.252:41451)
0927 15:11:55.555354 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.555359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.556619 (+ 1260us) server_negotiation.cc:407] Connection header received
0927 15:11:55.556916 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.556919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.556969 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.557047 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.557901 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.558661 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.559909 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.560204 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.560280 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.562166 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:55.562170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:55.562172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:55.562200 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:55.563837 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.564317 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.564321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.564323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.564408 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.564703 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.564707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.564708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.564957 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:55.565084 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:55.565445 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 15:11:55.565571 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 15:11:55.563144 30167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:11:55.566222 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.555954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50402 (local address 127.7.17.253:38813)
0927 15:11:55.556065 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.556069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.557201 (+ 1132us) server_negotiation.cc:407] Connection header received
0927 15:11:55.557368 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.557372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.557420 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.557470 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.558303 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.558978 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.559982 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.560348 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.560454 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.562355 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:55.562359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:55.562362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:55.562392 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:55.563985 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.564450 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.564453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.564455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.564541 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.564821 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.564854 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.564856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.565075 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:55.565185 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:55.565945 (+ 760us) server_negotiation.cc:299] Negotiation successful
0927 15:11:55.566069 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:11:55.567483 30167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:11:55.574286 30084 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d361afcc93fd4edcbf5c44821a8eba98 [term 1 FOLLOWER]: Refusing update from remote peer 7a28524343f7453fa37399ca23704bb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:55.575064 29963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:55.556661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54066 (local address 127.7.17.254:36247)
0927 15:11:55.556783 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.556787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.560881 (+ 4094us) server_negotiation.cc:407] Connection header received
0927 15:11:55.561086 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.561090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.561149 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.561229 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.566830 (+ 5601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.567624 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.568789 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:55.569172 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:55.569246 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:55.571116 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:11:55.571120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:11:55.571123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:11:55.571156 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:11:55.572922 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.573521 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.573525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.573527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.573623 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:11:55.573953 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:11:55.573957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:11:55.573959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:11:55.574222 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:11:55.574340 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:55.574730 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 15:11:55.574890 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:11:55.575294 30119 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d361afcc93fd4edcbf5c44821a8eba98" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:55.580673 30145 sys_catalog.cc:455] T 00000000000000000000000000000000 P d361afcc93fd4edcbf5c44821a8eba98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a28524343f7453fa37399ca23704bb6. Latest consensus state: current_term: 1 leader_uuid: "7a28524343f7453fa37399ca23704bb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a28524343f7453fa37399ca23704bb6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36247 } } peers { permanent_uuid: "275c0d89025d4256a6c1922e3f6ddb4f" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } } peers { permanent_uuid: "d361afcc93fd4edcbf5c44821a8eba98" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } } }
I20260927 15:11:55.580897 30145 sys_catalog.cc:458] T 00000000000000000000000000000000 P d361afcc93fd4edcbf5c44821a8eba98 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:11:55.581559 30008 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 275c0d89025d4256a6c1922e3f6ddb4f [term 1 FOLLOWER]: Refusing update from remote peer 7a28524343f7453fa37399ca23704bb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:11:55.582312 30157 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "275c0d89025d4256a6c1922e3f6ddb4f" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:11:55.586799 30133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 275c0d89025d4256a6c1922e3f6ddb4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a28524343f7453fa37399ca23704bb6. Latest consensus state: current_term: 1 leader_uuid: "7a28524343f7453fa37399ca23704bb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a28524343f7453fa37399ca23704bb6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36247 } } peers { permanent_uuid: "275c0d89025d4256a6c1922e3f6ddb4f" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } } peers { permanent_uuid: "d361afcc93fd4edcbf5c44821a8eba98" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } } }
I20260927 15:11:55.586968 30133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 275c0d89025d4256a6c1922e3f6ddb4f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:11:55.594739 30170 mvcc.cc:205] Tried to move back new op lower bound from 7333977766181564416 to 7333977766107246592. Current Snapshot: MvccSnapshot[applied={T|T < 7333977766181564416}]
I20260927 15:11:55.594939 30145 sys_catalog.cc:455] T 00000000000000000000000000000000 P d361afcc93fd4edcbf5c44821a8eba98 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a28524343f7453fa37399ca23704bb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a28524343f7453fa37399ca23704bb6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36247 } } peers { permanent_uuid: "275c0d89025d4256a6c1922e3f6ddb4f" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } } peers { permanent_uuid: "d361afcc93fd4edcbf5c44821a8eba98" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } } }
I20260927 15:11:55.595052 30145 sys_catalog.cc:458] T 00000000000000000000000000000000 P d361afcc93fd4edcbf5c44821a8eba98 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:11:55.596532 30119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a28524343f7453fa37399ca23704bb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a28524343f7453fa37399ca23704bb6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36247 } } peers { permanent_uuid: "275c0d89025d4256a6c1922e3f6ddb4f" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } } peers { permanent_uuid: "d361afcc93fd4edcbf5c44821a8eba98" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } } }
I20260927 15:11:55.596690 30119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:55.596889 30169 mvcc.cc:205] Tried to move back new op lower bound from 7333977766181564416 to 7333977766107246592. Current Snapshot: MvccSnapshot[applied={T|T < 7333977766181564416}]
I20260927 15:11:55.600128 30119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a28524343f7453fa37399ca23704bb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a28524343f7453fa37399ca23704bb6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36247 } } peers { permanent_uuid: "275c0d89025d4256a6c1922e3f6ddb4f" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } } peers { permanent_uuid: "d361afcc93fd4edcbf5c44821a8eba98" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } } }
I20260927 15:11:55.600275 30119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6 [sys.catalog]: This master's current role is: LEADER
I20260927 15:11:55.601688 30133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 275c0d89025d4256a6c1922e3f6ddb4f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a28524343f7453fa37399ca23704bb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a28524343f7453fa37399ca23704bb6" member_type: VOTER last_known_addr { host: "127.7.17.254" port: 36247 } } peers { permanent_uuid: "275c0d89025d4256a6c1922e3f6ddb4f" member_type: VOTER last_known_addr { host: "127.7.17.253" port: 38813 } } peers { permanent_uuid: "d361afcc93fd4edcbf5c44821a8eba98" member_type: VOTER last_known_addr { host: "127.7.17.252" port: 41451 } } }
I20260927 15:11:55.601816 30133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 275c0d89025d4256a6c1922e3f6ddb4f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:11:55.602456 30172 mvcc.cc:205] Tried to move back new op lower bound from 7333977766181564416 to 7333977766107246592. Current Snapshot: MvccSnapshot[applied={T|T < 7333977766181564416}]
I20260927 15:11:55.608183 30167 catalog_manager.cc:1347] Generated new cluster ID: c581b01d789547a0b5d60cbc4d3c87fd
I20260927 15:11:55.608281 30167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:11:55.663386 30167 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:11:55.664878 30167 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:11:55.684695 30167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a28524343f7453fa37399ca23704bb6: Generated new TSK 0
I20260927 15:11:55.685771 30167 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:11:55.723033 30116 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:55.720291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58752 (local address 127.7.17.252:41451)
0927 15:11:55.720430 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.720435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.721914 (+ 1479us) server_negotiation.cc:407] Connection header received
0927 15:11:55.722126 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.722129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.722181 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.722246 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.722894 (+ 648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58752: BlockingRecv error: recv got EOF from 127.0.0.1:58752 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 15:11:55.724671 30039 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:55.721270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50416 (local address 127.7.17.253:38813)
0927 15:11:55.721390 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.721395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.723733 (+ 2338us) server_negotiation.cc:407] Connection header received
0927 15:11:55.723909 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.723913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.723964 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.724013 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.724554 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50416: BlockingRecv error: recv got EOF from 127.0.0.1:50416 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 15:11:55.726385 29963 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:55.725348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54072 (local address 127.7.17.254:36247)
0927 15:11:55.725485 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.725490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.725561 (+ 71us) server_negotiation.cc:407] Connection header received
0927 15:11:55.725709 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.725712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.725765 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.725839 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.726260 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54072: BlockingRecv error: recv got EOF from 127.0.0.1:54072 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 15:11:55.731577 30039 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:55.729787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50424 (local address 127.7.17.253:38813)
0927 15:11:55.729901 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.729905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.730428 (+ 523us) server_negotiation.cc:407] Connection header received
0927 15:11:55.730685 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.730689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.730780 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.730844 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.731456 (+ 612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50424: BlockingRecv error: recv got EOF from 127.0.0.1:50424 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 15:11:55.737828 30039 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:55.735814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50428 (local address 127.7.17.253:38813)
0927 15:11:55.735954 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.735959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.736639 (+ 680us) server_negotiation.cc:407] Connection header received
0927 15:11:55.736941 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.736945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.737011 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.737073 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.737700 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50428: BlockingRecv error: recv got EOF from 127.0.0.1:50428 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 15:11:55.744441 30039 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:55.742603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50432 (local address 127.7.17.253:38813)
0927 15:11:55.742728 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.742733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.743346 (+ 613us) server_negotiation.cc:407] Connection header received
0927 15:11:55.743579 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.743582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.743644 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.743699 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.744298 (+ 599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50432: BlockingRecv error: recv got EOF from 127.0.0.1:50432 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 15:11:55.752936 30039 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:55.751119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.7.17.253:38813)
0927 15:11:55.751267 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.751272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.751854 (+ 582us) server_negotiation.cc:407] Connection header received
0927 15:11:55.752084 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.752088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.752149 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.752203 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.752782 (+ 579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50438: BlockingRecv error: recv got EOF from 127.0.0.1:50438 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 15:11:55.765489 30039 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:55.763829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50440 (local address 127.7.17.253:38813)
0927 15:11:55.763965 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.763970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.764617 (+ 647us) server_negotiation.cc:407] Connection header received
0927 15:11:55.764752 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.764756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.764806 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.764911 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.765367 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50440: BlockingRecv error: recv got EOF from 127.0.0.1:50440 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 15:11:55.784937 30039 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:55.783710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50442 (local address 127.7.17.253:38813)
0927 15:11:55.783849 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.783854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.784084 (+ 230us) server_negotiation.cc:407] Connection header received
0927 15:11:55.784228 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.784231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.784279 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.784346 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.784791 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50442: BlockingRecv error: recv got EOF from 127.0.0.1:50442 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 15:11:55.820294 30039 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:55.819152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50452 (local address 127.7.17.253:38813)
0927 15:11:55.819312 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.819318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.819511 (+ 193us) server_negotiation.cc:407] Connection header received
0927 15:11:55.819635 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.819637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.819685 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.819758 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.820168 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50452: BlockingRecv error: recv got EOF from 127.0.0.1:50452 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 15:11:55.887492 30039 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:55.886416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50464 (local address 127.7.17.253:38813)
0927 15:11:55.886542 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:55.886545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:55.886654 (+ 109us) server_negotiation.cc:407] Connection header received
0927 15:11:55.886809 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:55.886811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:55.886866 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:55.886951 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:55.887379 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50464: BlockingRecv error: recv got EOF from 127.0.0.1:50464 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 15:11:56.018714 30039 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:56.017703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50476 (local address 127.7.17.253:38813)
0927 15:11:56.017822 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:56.017826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:56.017934 (+ 108us) server_negotiation.cc:407] Connection header received
0927 15:11:56.018058 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:56.018060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:56.018121 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:56.018206 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:56.018609 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50476: BlockingRecv error: recv got EOF from 127.0.0.1:50476 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 15:11:56.277890 30039 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:11:56.276932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50482 (local address 127.7.17.253:38813)
0927 15:11:56.277053 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:56.277057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:56.277071 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:11:56.277197 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:56.277199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:56.277264 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:56.277344 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:11:56.277784 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50482: BlockingRecv error: recv got EOF from 127.0.0.1:50482 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:11:56.553668 30144 catalog_manager.cc:1259] Loaded cluster ID: c581b01d789547a0b5d60cbc4d3c87fd
I20260927 15:11:56.553804 30144 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 275c0d89025d4256a6c1922e3f6ddb4f: loading cluster ID for follower catalog manager: success
I20260927 15:11:56.555608 30144 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 275c0d89025d4256a6c1922e3f6ddb4f: acquiring CA information for follower catalog manager: success
I20260927 15:11:56.556975 30144 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 275c0d89025d4256a6c1922e3f6ddb4f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:11:56.565563 30158 catalog_manager.cc:1259] Loaded cluster ID: c581b01d789547a0b5d60cbc4d3c87fd
I20260927 15:11:56.565657 30158 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d361afcc93fd4edcbf5c44821a8eba98: loading cluster ID for follower catalog manager: success
I20260927 15:11:56.567409 30158 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d361afcc93fd4edcbf5c44821a8eba98: acquiring CA information for follower catalog manager: success
I20260927 15:11:56.568655 30158 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d361afcc93fd4edcbf5c44821a8eba98: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:11:56.796221 30241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:56.792071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.7.17.253:38813)
0927 15:11:56.792323 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:56.792328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:56.792401 (+ 73us) server_negotiation.cc:407] Connection header received
0927 15:11:56.792575 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:56.792577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:56.792709 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:56.792811 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:11:56.793172 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:56.793740 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:56.794665 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:56.794899 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:56.794992 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:56.795341 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:11:56.795819 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:11:56.795940 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:56.795993 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 15:11:56.796056 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":76,"threads_started":1}
I20260927 15:11:56.803193 30247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:56.799313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.7.17.252:41451)
0927 15:11:56.799561 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:56.799566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:56.799580 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:11:56.799679 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:56.799682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:56.799873 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:56.799990 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:11:56.800252 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:56.800783 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:56.801653 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:56.801892 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:56.801981 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:56.802296 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:11:56.802758 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:11:56.802876 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:56.802952 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 15:11:56.803016 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":68,"threads_started":1}
I20260927 15:11:56.810580 30241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:56.806334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50498 (local address 127.7.17.253:38813)
0927 15:11:56.806464 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:56.806468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:56.806695 (+ 227us) server_negotiation.cc:407] Connection header received
0927 15:11:56.806918 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:56.806921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:56.806978 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:56.807021 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:11:56.807393 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:56.808102 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:56.809470 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:56.809705 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:56.809795 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:56.810102 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:11:56.810176 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:11:56.810233 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:56.810382 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 15:11:56.810445 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:11:56.810804 30247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:56.806371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58776 (local address 127.7.17.252:41451)
0927 15:11:56.806497 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:56.806501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:56.807029 (+ 528us) server_negotiation.cc:407] Connection header received
0927 15:11:56.807112 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:56.807115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:56.807157 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:56.807214 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:11:56.807430 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:56.808003 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:56.809439 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:56.809780 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:56.809902 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:56.810341 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:11:56.810420 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:11:56.810481 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:56.810619 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 15:11:56.810677 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:11:56.811383 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:11:56.806659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54080 (local address 127.7.17.254:36247)
0927 15:11:56.806912 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 15:11:56.806917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:11:56.806929 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:11:56.807022 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:11:56.807024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:11:56.807193 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:11:56.807327 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:11:56.807657 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:56.808361 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:56.809479 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:11:56.809817 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:11:56.809902 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:11:56.810427 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:11:56.810798 (+ 371us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:11:56.810881 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:11:56.811168 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 15:11:56.811234 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":67,"threads_started":1}
I20260927 15:11:56.814229 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 29891
I20260927 15:11:56.820369 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 29966
I20260927 15:11:56.825966 7239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgkjf1n/build/debug/bin/kudu with pid 30041
2026-09-27T15:11:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3050 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6070 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (280805 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST